I updated an application from Django 4.2 to Django 5 on Monday (and then to 5.1 the next day). Since the upgrade on Monday morning, we’ve seen a significant performance decrease, specifically in rendering based on the data we collect using Scout APM. I have looked through the upgrade docs and I’m not seeing anything specific called out which seems like it would have this drastic of an impact (~180ms increase in mean response time and ~250ms increase in 95th percentile).
I’ve attached some screenshots of a day’s performance with Django 4 and a day with 5, as well as the past week performance of the page which receives the most traffic. You can see in that at ~8am on May 5 the rendering time for that view jumps up a lot.
I figure that there are many app specifics which would play into this and I’m happy to provide more specific details but I’m generally curious if there are specific changes made between Django 4 and 5 on template rendering which could be problematic for us?
Hi, just wanted to note here that I also noted a performance penalty in templates after upgrading from 4.2 to 5.2.
One big page went from 1.2s to 2s aprox when rendering about 3000 items in a loop.
Sentry said the average response time has increased about 300ms (same as author here).
At the same time I also upgrade Python from 3.8 to 3.12, Ubuntu from 20.04 to 24.04. So there were many changes and probably is not directly related to Django.
I saw that template performance in Django has only improved: airspeed velocity
I’m not able to go back to 4.2 and share changes, sorry.
Here’s an example of the view I specifically shared the screenshot of, but all customer facing pages on the site have the same jump. It is interesting that admin pages don’t seem to have changed in terms of rendering time. So, my guess is that there is something in the base template, context processors, or something else shared we do. All the customer facing pages have the jump in time. There are some somewhat expensive queries we do: looking up nearby shows / cities for users in middleware, fetching a list of ~200 cities to subscribe up updates about in the newsletter subscribe in the footer which is passed through a context provider.
The SQL query itself on the show details page pulls in a fair number of related records (information on ticket price and how many are left, the neighborhood, city, and info about the venue, etc.). Per the debug toolbar we render 7 templates and there are 9 context processors (8 without debug toolbar).
The view also displays different information based on whether the show is happening in the future, is sold out, is happening today, or was in the past.
We upgraded some other libraries alongside Django as well but did not upgrade Python. We’re on 3.10.
Can you isolate the “building the context” part of the view from “processing the template”? I realise that’s likely what Scout is doing, but it would be good to know that it’s a problem with Django rather than an instrumenting issue with Scout.
Thanks for sharing the profiler - it does seem to have some bugs with Django 5 and the full stack details don’t appear. I get pretty significant differences each page load and it’s hard to tell how accurate it is - DEBUG is true and I’ve set the cache to DummyCache but perhaps some of the template is cached still on reloads? I’ve tried to comment out certain lines and refreshing to profile parts of the template but they still seem to be included in the results.
That said, one thing which the profiler says takes a while is a for loop used to render the ticket options. Usually there is only ever 1 ticket, sometimes there are 2. Looking at the profiling for a couple other pages, it does seem like for loops over models pop up frequently. 2nd screenshot is the homepage. Is it possible this could actually be more related to slower SQL queries which are lazy evaluated?
Absolutely it could! QuerySets don’t execute SQL until you do something with them (for example, a for loop). Therefore, the time taken to do a for loop may be artificially increased by the cost of the SQL.
With that said, this has been the case in Django for a while (definitely 4.2, and well before that). So I’d be surprised if an update to 5.2 is changing that. Checking the query plan of the related query, the query itself, any caching you might have setup etc.
@nburt To get any answer, you’ll need to reduce it to the smallest example that demonstrates the performance change. Likely in constructing such an example you’ll discover the cause.
I’ll just add that since some folks mentioned noticing this coincident with a python 3.8->3.12 upgrade, that I noticed significant performance regressions in some of my tests python 3.11->3.12. They have improved somewhat 3.12->3.13.
If anyone is going to dig into this, I’d look at Django/python version combos.
I’m still noticing the issues - might start testing upgrading Python versions but so far I’m still stumped as to why Django 5 performance would be lower. I have made some improvements generally with some caching of some SQL queries made in middleware but that seems separate. While they have reduced overall page loads, that’s all from middleware in Scout and template rendering still is high.
With debug on in logs, I do notice a lot of context lookup related errors, e.g. VariableDoesNotExist(
django.template.base.VariableDoesNotExist: Failed lookup for key [city]. Do those types of errors contribute significantly to rendering times to attempt and lookup the variable or is that not really something needing fixing?
I also upgraded Whitenoise to latest - realized we didn’t upgrade that to support 5.1 and thought that could have led to some of our static assets not getting cached properly but doesn’t seem to have fixed things.
The other thing I was wondering if template caching may have changed. This is what we’ve had:
