regroup template tag breaks when upgrading from Django 2 to 4

Using Django Templates & Frontend
1

Hi everyone,

I’ve encountered an issue with a test case that passes in Django version 2.2 but fails when I upgrade to Django 4.2. The test verifies correct regrouping of data in a Django template. Here’s a brief overview of the application, test setup and problem:

  • Application overview: The application is an open-source foodbank campaign management system, the source is available on GitHub at mrts/foodbank-campaign. It’s designed to facilitate the organization and management of foodbank food collection campaigns, including managing shop locations where the food is collected and scheduling volunteer shifts in these locations.
  • Test setup: The test involves creating a Campaign and related CampaignLocationShift objects and using a Django template to regroup and render the data into a JavaScript variable shopsAndShifts. The template uses the {% regroup %} tag to organize location shifts by district and then by location. The code is here: campaigns/test.py.
  • Problem: Upon upgrading to Django 4.2, this test case fails. It seems the {% regroup %} tag or related queryset handling is different. Here’s the failure:
    AssertionError: var shopsAndShifts = { 3: [ { shop: 'Rocca Al Mare Prisma (Tallinn)', [truncated]... != var shopsAndShifts = { 3: [ { shop: 'Haabersti Rimi', shifts: [ { pk: [truncated]...
    In the Django 4.2 version the data is messed up so that the resulting JavaScript variable shopsAndShifts contains only a single location after being interpreted by the browser. Here’s how it is actually used in the application.

Question: Has anyone else encountered similar issues with regrouping when upgrading to Django 4+? Should I report this as a bug?

Django versions:

  • Django version before upgrade: 2.2.28
  • Django version after upgrade: 4.2.11

The template code for reference:

    {% regroup locations_and_shifts by location.district as district_list %}
    var shopsAndShifts = {
    {% for district in district_list %}
    {{ district.grouper.id }}: [
        {% regroup district.list by location as location_shifts %}
        {% for location in location_shifts %}
          {
            shop: '{{ location.grouper.name }}',
            shifts: [
            {% for shift in location.list %}
              {
                pk: '{{ shift.pk }}',
                when: '{{ shift.day }} {{ shift.start }} - {{ shift.end }}',
                freePlaces: {{ shift.free_places|default_if_none:shift.total_places }}
              }{% if not forloop.last %},{% endif %}
            {% endfor %}
            ]
          }{% if not forloop.last %},{% endif %}
        {% endfor %}
      ]{% if not forloop.last %},{% endif %}
    {% endfor %}
    };

I’ve already reviewed the Django release notes and documentation but haven’t pinpointed the exact cause of this issue. Thank you in advance for any insights or advice you can provide!

2

Hi,

You said that only a single location is rendered in the resulting javascript, but the test result doesn’t show that. How did you make this assumption ?

The only thing the failing test shows is that the begining of the resulting javascript does not contain the same location, which may happen because of a different ordering of the queryset.

Also, the queryset for locations_and_shifts has no ordering, which can be a problem when using the regroup tag has, as stated in its documentation, it relies on the ordering for grouping items.

To go further in debug, I suggest you first check the full dump (e.g. by adding a print statement) of the rendered javascript to ensure if it only contains partial elements or all elements in a different order.

I would also dump the queryset generated in the view to ensure data it contains are the same between the 2 django versions.

In both cases (different ordering or different data in queryset dump) , regroup tag is not at fault and the error is to be searched in how the queryset is built.

4

Sorry, my assumption about not having ordering is not correct, as an ordering Meta is specified in the CampaignLocationShift model.

However checking the queryset would still be an indication on whether regroup is at fault or not.

The first location being returned in the Bad result being the CampaignLocationShift with the lowest id, I suspect the ordering by location__district__name, location__name does not apply (correctly) on queryset. You may also check the differences in the sql issued for both versions of django.

The change in the queryset order may be because of the annotation for free_places which deactivates ordering: see #32811 (Annotate removes Meta.ordering) – Django. This is intended behaviour starting from Django 3.1.
You may add an explicit order_by() to your queryset to recover the Django 2 behaviour.

5

Thanks a lot for the tip! Yes, #32811 was the culprit, the following change fixed it:

--- a/src/campaigns/models.py
+++ b/src/campaigns/models.py
@@ -78,7 +78,10 @@ class CampaignLocationShiftManager(models.Manager):
         qs = super().get_queryset()
         return qs.annotate(free_places=F('total_places') -
                 Sum('volunteers__participant_count',
-                    output_field=IntegerField()))
+                    output_field=IntegerField())) \
+                        .order_by(*CampaignLocationShift._meta.ordering)