fix: serializer can't bypass manager for m2m

I made a fix on serializers' for nested field because we can't use manager.remove(item) when manager is an intermediary model and not the manager itself.