Need Help Fixing Many-to-Many Query and Template Loop

In this case, it would be multiple objects, along with its related collector objects based on prefetch_related connecting those related objects through the “collector” field.

Or of course it could be a single collection object and its related collector objects, which could be a single collector or multiple.

It’s always a QuerySet, consisting of one or more instances of “Collection” objects. (Each of those Collection objects has a reference to one or more Collector objects, but those Collector objects are not elements within that QuerySet.

So, if you have a Query set, how do you access individual Collection objects within the template?

Well, we set up a loop, and based on my limited experience, this might look something like …

collection.collector.all - this I believe is a set of collection objects (one or more) and all related collector objects (also one or more). It seems like getting to collector’s attributes should be “collection.name.” Then working down the hierarchy, to access collector object attributes, you would do, “collection.collector.inst_main_name.” But this is where, clearly, I’m out in the weeds.

Nope, you’re jumping ahead.

To reference an individual Collection in the QuerySet, you iterate over the QuerySet.
e.g.
{% for a_collection in collection %}

creates an iteration variable named a_collection containing a reference to a single Collection object.

Now that you have a reference to a single Collection object named a_collection, how do your reference the set of Collector objects associated with that one Collection?

I understand this part for sure. But to access the related Collector objects, I think you have to go through the many-to-many field, collector, so in this case, a_collection.collector.

Right, but a_collection.collector is a “related field manager”. To reference the set of Collector objects, that’s where you use a_collection.collector.all. So a_collection.collector.all is not an instance of a collector. It’s a set of all collector objects associated with that specific a_collection object.

Hey Ken, back with you on this vexing problem after spending some time in the admin area. Had great success there.

So what I’m not seeing is how this problem is any different from the following view and template code, which works. I do understand that in the following case, we are retrieving a single collection object and its associated collectors, which could be one or more.

def collection_detail(request, collection_id):
    collection = Collection.objects.prefetch_related('collector').get(pk=collection_id)
    template = 'collections_app/collection_detail.html'
    context = {'collection': collection}
    return render(request, template, context)

{% for collector in collection.collector.all %}
<div class="flex-container-body">
<div style="flex: 0 0 300px">
<h2>Collector Name</h2>
{% if collector.person_last_name %}
    <p><a href="">{{ collector.person_first_name }} {{ collector.person_last_name }}</a></p>
    <h3>Person Type(s)</h3>
        <ul class="body">
        {% for person_type in collector.person_type.all %}
        <li><a href="">{{ person_type }}</a></li>
        {% endfor %}
        </ul>
{% else %}
    <a href="">{{ collector.inst_main_name }}<br>{{ collector.inst_sub_name }}</a></p>
    <h3>Institution Type(s)</h3>
    <ul class="body">
    {% for inst_type in collector.inst_type.all %}
        <li><a href="">{{ inst_type }}</a></li>
   {% endfor %}
    </ul>
 {% endif %}
</div>

It’s not any different at all, and that’s the point.

In this example you just posted, you’re iterating over the set of Collector objects from a single collection item using the collector.all attributes, which is exactly what you want to do in your previous case

So here, once again is the template in question. What am I missing? Ken, thanks again for your persistence and patience.

<ul class="body">
    {% for collector in collection.collector.all %} #loop through collector objects gathered from prefetch_related
        <li><a href="/collections/{{ collection.id }}"> #grab collection id
            {{ collector.inst_main_name }} #grab related collector object's field
            "inst_main_name"
            {% if collector.inst_sub_name %}, {% endif %} #if related collector object
            has field "inst_sub_name", add a comma
            {{ collector.inst_sub_name }} #grab related collector object's field
            "inst_sub_name"
            {{ collector.person_last_name }} #grab related collector object's field
            "person_last_name"
            {% if collector.person_last_name %}, {% endif %} #if related collector
            object has field "person_last_name", add a comma
            {{ collector.person_first_name }} &#8226; {{ collection.name }}</a></li>
        #grab related collector object field "person_last_name", html entity, then collection object
        field "name"
    {% endfor %}
</ul>

On the second line -
{% for collector in collection.collector.all %}
What is the data type of collection?

How is that different from the data type of collection in the previous example?

Now are you beginning to see that naming them the same is causing confusion?

I don’t mean to butt in, it looks like you’re getting there anyway, but maybe a different perspective with an analogy might help.

Analogy setup
Say you have five hospitals, ‘Hospital A’, ‘Hospital B’, … ‘Hospital E’. They have the primary keys 1, 2, 3… 5.

Each hospital has different wards. Hospital A has the wards ‘Ward Foo’, ‘Ward Bar’, and ‘Ward Qux’. They have primary keys 1, 2, 3.

Each ward has one head nurse:
‘Ward Foo’: ‘Nurse Rogers’
‘Ward Bar’: ‘Nurse Lovelace’
‘Ward Qux’: ‘Nurse Turing’

Analogy to what you are doing
hospitals = {all hospitals that have primary key 1}
Note that here, you get a set of hospitals, it just happens to be a set consisting of just the one hospital, i. e. {‘Hospital A’}

for ward in hospitals.wards.all:
    print(ward.head_nurse)

This doesn’t work. This is because the set of hospitals, hospitals, does not have a ‘wards’ attribute. It’s just a set. Each hospital has a ‘wards’ attribute of course, but the set of hospitals doesn’t have anything like that.

If you wanted to, though it’s unnecessarily complicated, you could use the set like this:

for hospital in hospitals:
    for ward in hospital.wards.all:
        print(ward.head_nurse)

This works because, again, hospitals is a set (it doesn’t matter that it’s a set consisting of just one element - it’s still a set), which you can loop over.

Analogy to what you want to do
hospital = hospital that has primary key 1
Here, you fetch one specific hospital, not as part of a set, but as an object/instance ‘in its own right’.

If you now do

for ward in hospital.wards.all:
    print(ward.head_nurse)

This works! Since the hospital object/instance (which is not a set) does have an attribute ‘wards’.

So when you do:

collection = Collection.objects.filter(subject_city=id).prefetch_related('collector').all()

It would actually make more sense to change the variable name like this:

collections = Collection.objects.filter(subject_city=id).prefetch_related('collector').all()

(or you could call the variable collections_set). This is because you are asking Django: "Please give me the (query)set consisting of all Collection instances whose subject_city is id". The .all() part is analogous to the ‘all’ in this sentence.

Can you see what’s different then when you use .get here?

collection = Collection.objects.prefetch_related('collector').get(pk=collection_id)

Here’s an example of what happens in the shell (python manage.py shell), where cion1 is a Collection object:

>>> cion1.collector.filter(pk=3)
# outputs: <QuerySet [<Collector: Collector object (3)>]>

>>> cion1.collector.get(pk=3)
# outputs: <Collector: Collector object (3)>

In the first case, I asked for a set consisting of all things that fulfill a criterion. In the second case, I asked for one particular thing that fulfills a criterion.

But now I’m just getting repetitive :stuck_out_tongue: I hope this helps! (also, very nice site layout/theme IMO)

(@KenWhitesell I hope this isn’t too much of giving a fish, distracting from your teaching how to fish)

1 Like

@datalowe Nope, no problem at all. A different perspective or approach is rarely a bad thing.

Datalowe, thanks mucho for taking the time for sharing that great analogy. And, Ken, I don’t know who you are, but I continue to appreciate your classroom.

I think all I needed to understand is that we must take a hierarchical approach to looping through these sets. First, we break up the set’s first layer with the first loop, which is collection, then we can move on to the next loop, which gets us to collector. I think I always understood the difference in queries that get sets and a get query, which give you a single instance. However, I wasn’t quite getting how to do a nested loop … I had not done one of those before. You said the method you shared was unnecessarily complicated, but maybe I missed the more elegant solution. I suspect we can work through those layers with a single hierarchal statement.

Also, I really appreciate the boost about the layout. It’s not quite there, and as I add stuff, I find I need to tweak it to accommodate new features. But I’m generally pleased. Y’all keep in mind that this is my first time out using Python, much less Django. However, victories like this keep me going. This directory has been my quarantine project.

<ul class="body">
    {% for collection in collections %}
        {% for collector in collection.collector.all %}
        <li>
        <a href="/collections/{{ collection.id }}">
        {{ collector.inst_main_name }}{% if collector.inst_sub_name %}, {% endif %}
        {{ collector.inst_sub_name }}
        {{ collector.person_last_name }}{% if collector.person_last_name %}, {% endif %}
        {{ collector.person_first_name }} &#8226; {{ collection.name }}</a>
        </li>
        {% endfor %}
    {% endfor %}
</ul>

Happy hour, anyone? I’m buying!

I think I always understood the difference in queries that get sets and a get query, which give you a single instance. However, I wasn’t quite getting how to do a nested loop … I had not done one of those before.

Ah, so when you used Collection.objects.filter(subject_city=id).prefetch_related('collector').all(), you did in fact want a set of potentially multiple collections that have the same “subject_city” value? That makes sense then, I had misunderstood what it was you were aiming for. I’m glad my post helped anyway.

However, I wasn’t quite getting how to do a nested loop … I had not done one of those before.

While Django is great, I think, for making practical use of Python skills, it’s also worth focusing on learning Python itself now and then. There are more parts to keep track of with Django, so learning core skills can become more complicated. It’s often easier to go the other way around - learning core skills and then combining them together (e. g. in Django). Of course, running into the real problems that you need the tools for first (e. g. with Django) can help give you context and motivation. But it’s a good habit to dive deeper into what’s going on when you don’t understand something, because over time it helps you amass background knowledge that lets you connect the dots and learn things more quickly (at least that’s my experience so far). Here’s a book that helped me a lot, linking specifically to a section about nested for loops: http://openbookproject.net/thinkcs/python/english3e/iteration.html#nested-loops-for-nested-data

datalowe in Stockholm, how exotic btw, compared to Fort Worth, Texas …

I’m changing threads a bit, probably not a good idea, but I’m on to the next challenge.

I have the following field in my model for collection:

image = models.ForeignKey('CollectionImage',
                              related_name='collections',
                              verbose_name=u'Collection Images',
                              on_delete=models.CASCADE,
                              blank=True,
                              null=True,
                              help_text='Upload images showing example material from files, storage systems in use, or documents relating to the collection.')

And its related class:

class CollectionImage(models.Model):
    image = models.ImageField('Add Image',
                              upload_to='collection/images/',
                              help_text='Upload images showing example material from files '
                                        'and/or storage systems in use.')

    def __str__(self):
        return 'Image'

    class Meta:
        verbose_name = 'Image'
        verbose_name_plural = 'Images'

I have a view that looks like this:

def collection_detail(request, collection_id):
    collection = Collection.objects.prefetch_related('collector').select_related('image').get(
        pk=collection_id)
    template = 'collections_app/collection_detail.html'
    context = {'collection': collection}
    return render(request, template, context)

I’m having trouble accessing the uploaded images uploaded related to a collection, which I confirm are there. I worry now that the data model is not correct. Yet it makes sense to me to have image upload appear when one enters a new collection, rather than have the foreign field appear in the CollectionImage model. Am Do I have a good foundation? I’m just having trouble getting those images to appear in the template.

Right now my template is thus:

{% if collection.image %}


    {% for image in collection.image.all %}
  • Collection Image

  • {% endfor %}

{% endif %}

Sharing admin view of image field:

in Stockholm, how exotic btw

That is a very American thing to say IMO. Please don’t think of us or our country as “exotic”, instead simply as people who happen to live in a different place :slight_smile: I reckon you don’t speak German? Otherwise this song was, in a way, basically made for your statement https://www.youtube.com/watch?v=pLl6vpopJAk

Anyway:

  1. As Ken mentioned earlier, everything becomes more confusing when you use the same name to refer to different things. You now have the image ForeignKey of Collection, as well as the image ImageField of CollectionImage. This means that if you want to use a Collection instance as a starting point and refer to the image ImageField of CollectionImage, you have to write collectioninstance.image.image. It would probably be best if you could use alternative names.

  2. Did you define the MEDIA_URL and MEDIA_ROOT variables in your project’s settings.py? Otherwise you probably want to do that. In addition, you need to add some things to your project-level urls.py: https://docs.djangoproject.com/en/3.0/howto/static-files/#serving-files-uploaded-by-a-user-during-development

  3. In your template, you write {% for image in collection.image.all %}. But collection.image is not a set of things. This is because you used a ForeignKey field to define the relationship between Collection and CollectionImage. When using a foreign key, the database/django can only link each Collection instance (/entry in the database) to one CollectionImage instance. So there is no all() method to use with collection.image, and there isn’t anything to loop over.
    What you can do instead to refer to the url for the linked CollectionImage instance’s image (again, note that the naming makes things confusing) is e. g.:

{% if collection.image %}
<h2>Image</h2>
<img src="{{ collection.image.image.url }}">
{% endif %}

I haven’t done much with media files, so anyone who sees this and can correct/clarify things, please do so. I hope it’s of some help either way.

Aw, the pitfalls of being a newbie. Thank all, especially you, datalowe, for taking the time to explain things so clearly. I call you on the exotic comment because it was meant in a flattering way. I use that word a lot for things outside my normal realm, like Django database work! :wink:

So, I suspect, and what does not seem to be made clear in the documentation relating to foreign fields is the repercussion of where you place them, is that I placed foreign key on the wrong side. Somewhere I read that the foreign key goes on the “many” side. What I want is for each collection record to display multiple images in a lovely and CSS flexbox. I get confused about this because it seems you would want the image upload field to show in the collection model, not in the CollectionImage model. That workflow seems more intuitive, at least in the admin area. But maybe these things get sorted out when you build the user facing input forms. So what happens if I flip the foreign key? Will it break my database? How does it change the query and template code?

BTW, I now have a lovely, yet single, image displaying with your help. And I understand your instruction about getting it to show.

Is there any chance we could have a Zoom chat about my project? I would be more than happy to pay you!