UNNEST bulk_create optimisation causes integer overflow for models referencing django_admin_log

Django Internals ORM
1

The UNNEST bulk_create optimisation introduced in Django 5.2 (ticket #35936) generates explicit PostgreSQL type casts derived from field.db_type().

LogEntry does not declare an explicit id field. Its primary key type is inherited from the admin app’s AppConfig, which hardcodes default_auto_field = “django.db.models.AutoField” (source). This override was added in commit b5e12d49] to prevent unwanted migration generation when projects set DEFAULT_AUTO_FIELD = BigAutoField.

When calling LogEntryTracing.objects.bulk_create(…), Django 5.2 generates:


INSERT INTO log_entry_tracing (log_entry_id, trace_id)

SELECT * FROM UNNEST(

(ARRAY[3184205939, 3184205940])::integer[],

(ARRAY['trace_a', 'trace_b'])::varchar[]

)

The ::integer[] cast comes from LogEntry.id.db_type() returning “integer” (via AutoField). PostgreSQL rejects the cast because 3184205939 > 2,147,483,647.

This also affects LogEntry.objects.bulk_create() indirectly – while the auto-generated id is omitted from the INSERT (so no cast on id itself), any model holding a FK to LogEntry.id will get the ::integer[] cast on its FK column.

Before 5.2, this was harmless – VALUES (%s, %s, …) doesn’t type-cast parameters. In 5.2, the UNNEST path in django/db/backends/postgresql/compiler.py does:


db_types = [field.db_type(self.connection).split(“(”)[0] for field in fields]

# produces ::integer[] for AutoField

When calling LogEntryTracing.objects.bulk_create(...), Django 5.2 generates:


INSERT INTO log_entry_tracing (log_entry_id, trace_id)

SELECT * FROM UNNEST(

    (ARRAY[3184205939, 3184205940])::integer[],

    (ARRAY['trace_a', 'trace_b'])::varchar[]

)

The ::integer[] cast comes from LogEntry.id.db_type() returning "integer" (via AutoField). PostgreSQL rejects the cast because 3184205939 > 2,147,483,647.

This also affects LogEntry.objects.bulk_create() indirectly – while the auto-generated id is omitted from the INSERT (so no cast on id itself), any model holding a FK to LogEntry.id will get the ::integer[] cast on its FK column.

DEFAULT_AUTO_FIELD is not used

Projects can set DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField" globally, but Options._get_default_pk_class() checks app_config.default_auto_field first:


# django/db/models/options.py

def _get_default_pk_class(self):

    pk_class_path = getattr(

self.app_config,

"default_auto_field",

        settings.DEFAULT_AUTO_FIELD,

)

Since SimpleAdminConfig explicitly sets default_auto_field = "django.db.models.AutoField", the global setting is ignored for LogEntry.

Proposed change

Add an explicit BigAutoField primary key to LogEntry and ship a migration. This is a one-line model change plus a migration file:

Model change (django/contrib/admin/models.py):


class LogEntry(models.Model):

id = models.BigAutoField(primary_key=True)   # <-- add this

    action_time = models.DateTimeField(...)

Alternative proposal

Remove the default_auto_field from SimpleAdminConfig so that DEFAULT_AUTO_FIELD takes precedence

class SimpleAdminConfig(AppConfig):
    """Simple AppConfig which does not do automatic discovery."""

    # default_auto_field = "django.db.models.AutoField"
1 Like
2

Hello @frepettov0, thank you for the detailed thread.

I would argue that the problem you are facing here has little to do with the bulk_create UNNEST optimization as Django should be able to assume that the model definition it operates from is coherent with the database it interacts with.

You can expect many ORM things to break if your model definitions don’t match your table ones so I don’t see an issue with bulk_create here; it seems you are more interested in allowing admin models to use a BigAutoField primary key.

Add an explicit BigAutoField primary key to LogEntry and ship a migration.

This would incur a primary key alteration migration which is an operation that locks tables on most backends.

This was an option we opted against as the default because in cases where tables are small, and the migration is relatively safe, they would get very little benefit from moving to BigAutoField in the first place but in the case of large tables, which would benefit the most, the migration would need to be performed with care so introducing some friction seems warranted.

What I suggest you do is create your own bigauto_admin module that does the following

# bigauto_admin/apps.py
from django.contrib.admin.apps import AdminConfig

class BigAutoAdminConfig(AdminConfig):
     name = "django.contrib.admin"
     label = "admin"

You then remove "django.contrib.admin" from your INSTALLED_APPS and add "bigauto_admin.apps.BigAutoAdminConfig" and make sure to set MIGRATION_MODULES["admin"] = "bigauto_admin.migrations".

You then run makemigrations to generate the migrations as they are defined by django.contrib.admin in core.

You then set BigAutoAdminConfig.default_auto_field = "django.db.models. BigAutoField", run makemigrations again, and voilà you’ve got the migration override you were looking for and more importantly isinstance(LogEntry.id, BigAutoField) from now on.

I’m not sure we’ve documentd a pattern for replacing third-party app migrations before but that’s one I’ve used successfully in the past.

3

Hi @charettes ,

Thanks to you for the quick reply!

Apologies for poorly explaining what we wanted, but you got it right :grinning_face_with_smiling_eyes:

I am happy the solution you mentioned was one in our plate already, so we will go in that direction. I was unsure if there was a bug Django in LogEntry between DEFAULT_AUTO_FIELD and django.db.models.AutoField (there isn’t)

Again, thank you!

1 Like
4

Hi @charettes, as you rightly said, setting BigAutoAdminConfig.default_auto_field = "django.db.models.BigAutoField" generated a new migration 0004_alter_logentry_id with the expected override.

My concern is, what would happen when Django decides to make some changes in LogEntry in newer releases and it results in a new migration 0004_xxxxx.

How do we patch this new migration on our project if other apps in the project build a dependency on 0004_alter_logentry_id?

This might be a naive question but please bear with me, its still my early days in the Django world :grinning_face:.

5

Assuming you set MIGRATION_MODULES adequately (which your question about an existing 0004_xxxxx makes me doubt you did) your special purposed Django app migration module should completely override the django.contrib.admin ones meaning that if Django ships model changes over there you should only have to run makemigrations and migrate again.

6

I missed to provide some additional context.

We already have some migrations that depends on the 0003_logentry_add_action_flag_choices present in django.contrib.admin. This is because some models have OneToOneField relationship with the LogEntry model.

So I had to copy the existing migration files in the new bigauto_admin app.