Automatic raw_id_fields in the admin for large queries

Django Internals ORM
1

Would it be possible to put in some kind of automatic safeguard in the Django admin for foreign keys to prevent the server/browser from crashing when there are too many items in the foreign key table? It seems like this has been brought up a few times in tickets 17881, 25114, and I’m sure there are others. However not much progress has been made on this issue in the past 12 years or so.

I ran into this issue today with a model that has grown over time. It started out having fewer than a hundred items in the table. However it has now grown to the point where it will crash the admin with over 100k items due to a change in functionality over time. It is used frequently across many other models, so now a good chunk of our admin is no longer functional without having to add the field to the raw_id_fields lists. While this change does not take long to make, it is a frequent footgun that has caused many issues.

I may be naive, but it seems like a check could be added that would run a count call on the queryset if the item is a foreign key field and not in the raw_id_fields list. If the count is above some set value that is known to cause issues with performance and/or usability, then it would fall back to a raw id field. This could be controlled by a setting value to be a bit more flexible.

Is this something that is possible to change in Django core, or are there other use cases that I’m not considering?

2

I’ve seen this problem recur on many projects too, and I would like to see some kind of solution. It is tricky to do the right thing in every case though.

The proposal to automatically use raw ID fields for large tables sounds reasonable. But we must remember that counting a table is still expensive. (Databases have to lock each row they touch to ensure an accurate count.) Also, counts can vary from request to request.

Perhaps we could use an opt-in limit instead? That is, all FK queries would limit to, say, 101 results. If there are 100 or fewer, render as normal. Otherwise, if there are 101, render the field as a raw ID one. (This “fetch N+1” trick is used already in QuerySet.objects.get().)

We’d still need to deal with field type changing on a per-request basis, especially between loading a changeform view and submitting its form.

1 Like