Query with ManyToManyField

Hello everyone, I’m a Django newbie so I’m sorry if this question has already been asked.
I tried to find an answer to my doubts, but I couldn’t.

I have two models:

#users/models.py

class CustomUser(AbstractUser):
    username = models.CharField(max_length=255, unique=True)
    email = models.CharField(max_length=255, unique=True)
    password = models.CharField(max_length=255)
    favorites = models.ManyToManyField(Activities)

#activities/models.py

class ActivityTypes(IntEnum):
    LEISURE = 0
    PHYSICAL = 1
    MENTAL = 2
    SOCIAL = 3

    @classmethod
    def choices(cls):
        return [(key.value, key.name) for key in cls]

class Activities(models.Model):
    name = models.CharField(max_length=255, unique=True)
    type = models.IntegerField(choices=ActivityTypes.choices(), default=ActivityTypes.LEISURE)
    solo = models.BooleanField(default=False)

And these are the serializers:

#users/serializers.py

class UserSerializer(serializers.ModelSerializer):
    favorites = ActivitiesSerializer(many=True, read_only=True)

    class Meta:
        model = CustomUser
        fields = ['id', 'username', 'email', 'password', 'favorites']
        extra_kwargs = {
            'password': {'write_only': True}
        }

    def create(self, validated_data):
        password = validated_data.pop('password', None)
        instance = self.Meta.model(**validated_data)
        if password is not None:
            instance.set_password(password)
        instance.save()
        return instance

#activities/serializers.py

class ActivitiesSerializer(serializers.ModelSerializer):
    class Meta:
        model = Activities
        fields = ('id', 'name', 'type', 'solo')

I’m trying to write a view to basically retrieve the activities of a certain type and the users that have those activities as favorites and return this result in JSON.
In sql it would be something like this (assuming that the through table is called Favorites):

select Activities.name, Activities.type, CustomUser.username, CustomUser.email
from Activities join Favorites on Activities.id = Favorites.activity
join CustomUser on Favorites.user = CustomUser.id
where type = "...";

By the way, I run this same query (adapted obviously) on the sqlite database I’m using and it returns exactly the data I need, so I would say that the models are somehow correct.

As you can assume, I’m not having much success in writing such query in Django.
I’m not even sure that my serializers are right for the job.

I was able to retrieve the users and their favorite activities, like so:

users = CustomUser.objects.exclude(favorites=None).prefetch_related('favorites')
serializer = UserSerializer(users, many=True)
return JsonResponse(serializer.data, safe=False)

but not the other way around (the activities and the users that have them as favorites).

So, how can I solve my problem?

Thanks in advance to anyone who will answer!

ManyToMany relationships are handled by a custom manager created for that purpose.

See the docs (and examples) at Many-to-many relationships | Django documentation | Django to get started.

I sort of solved the problem creating manually the results I needed.

I created a function that return a dictionary starting from an activity and a user that has that activity as favorite:

def createResponse(activity, user):
    output = {}
    output['name'] = activity.name
    output['type'] = activity.type

    if(user):
        output['username'] = user.username
        output['email'] = user.email
    
    return output

And this is a snippet of the view that uses it:

activities = Activities.objects.filter(type=type, solo=solo)

users = CustomUser.objects.exclude(favorites=None).prefetch_related('favorites')

output = []

for i in range(len(activities)):
    users = users.filter(favorites__id=activities[i].id)
    if len(users) > 0:
        for j in range(len(users)):
            output.append(createDict(activities[i], users[j]))
    else:
        output.append(createDict(activities[i], None))

return JsonResponse(output, safe=False)

I suspect that there is a better way to do this, but I have not yet been able to find it, even after reading the linked documentations.

Thanks anyway @KenWhitesell for the answer.

Why don’t you show us what you’ve tried? Who knows, you may be very close to the appropriate solution.

Why don’t you show us what you’ve tried? Who knows, you may be very close to the appropriate solution.

I’m sorry, what do you mean? I already reported my solution in my previous reply.

You wrote (and I quoted):

Yes, there is a significantly better way to do this. How it’s done is described in the docs, with examples.

If you tried to do what the docs show about querying a many-to-many relationship “backwards”, and were unable to get it to work, we may be able to help you with that. The best starting point for providing that assistance would be to see one of your attempts to do it, and we can work forward from there.

Ok, now it’s clear what you meant, thank you for clarifying.

Tell me if I’m wrong but from what I can see in the docs there are examples on how to do reverse queries filtering on the attributes of the model that has the m2m field.
But there is no example showing how to retrieve data from both models at the same time and that’s what I need.

Now, I would gladly show you one of my attempts in trying to do what the docs show, but the fact is that I simply don’t know how to attempt what I’m trying to do using the Django syntax, because from the examples I can’t understand how to do it.
If I use the syntaxes showed in the docs I obtain information about one model or the other and I can’t reconstruct the information composed by both models, unless I do it manually as I showed previously.

If you can point me to the right example or show me the syntax to obtain a join of both the models, keeping the results of both, I would really appreciate it.

I guess I’m not understanding what you mean by “retrieve data from both models at the same time”.

You’re retrieving data from one model, and associated with it, the data from the other.

To use your example:
CustomUser has an M2M relationship with Activities by the name favorites.

CustomUser.objects.all() returns a queryset containing all the elements of CustomUser.

When you’re iterating over that queryset, you can access all the favorites for a user using that name. For example if custom_user is an instance of CustomUser, then all the related Activities would be custom_user.favorites.all()

The term favorites in that expression is known as the RelatedManager. It’s the object that provides access to those related models.

It’s probably my fault that I can’t explain myself properly. Unfortunately english is not my main language.
Or maybe I’m just thinking in SQL when I should think in Django :thinking:.

Thanks for the example but the thing I can’t understand is how to go backwards.

Starting from Activities.objects.all() for example, I’d like to iterate on that queryset and for every activity retrieve, if they exists, all the related CustomUser(s).

So you have two choices when accessing the reverse relationship.

First, you can access the reverse related manager using the “_set” suffix on the model name as described at Following relationships backward.

Also see the examples at ManyToMany Relationships starting at the text:

Article objects have access to their related Publication objects:

along with the text at RelatedManager

Or, if you wish to be more explicit about it, you can specify the related_name parameter in your field definition.

1 Like

Thank you @KenWhitesell !!!

I finally understood how to do it, and it was a lot easier than I thought.
The answer was right in front of my face and I didn’t see it.
I should read more carefully the documentations!

Here’s a snippet of my updated view:

activities = Activities.objects.filter(type=type, solo=solo)

output = []

for i in range(len(activities)):
    output.append(createResponse(activities[i], activities[i].customuser_set.all()))

return JsonResponse(output, safe=False)

And here’s the function that creates the response:

def createResponse(activity, users):
    output = {}
    output['name'] = activity.name
    output['type'] = activity.type
    output['users'] = {}

    if(len(users) > 0):
        for i in range(len(users)):
            output['users']['user' + str(i)] = {}
            output['users']['user' + str(i)]['username'] = users[i].username
            output['users']['user' + str(i)]['email'] = users[i].email
    
    return output

Again, thank you @KenWhitesell, you’re a godsend!

I’ll add one more side note here - you don’t need that initial if statement. If len(users) is 0, the loop executes 0 times (meaning, it doesn’t).

1 Like

You’re right, thanks!