KeyError when migrating backward with a squashed migration includes RemoveField operation

Django Internals ORM
1

Hi,
I thinks there is a bug in django in that particular conditions.

When an app has a squashed migration which include a RemoveField operation and you try to migrate backward on a migration elsewhere (in the app or in another) it raise a KeyError on the removed field.

Steps to reproduce :

  1. Create Model
class Animal(models.Model):
    field_a = models.CharField("A", max_length=50, default="", blank=True)
    field_b = models.CharField("B", max_length=50, default="", blank=True)
  1. Make migrations makemigrations => 0001_initial.py
  2. Remove field B
class Animal(models.Model):
    field_a = models.CharField("A", max_length=50, default="", blank=True)
  1. Make migrations makemigrations => 0002_remove_animal_field_b.py
  2. Add field C
class Animal(models.Model):
    field_a = models.CharField("A", max_length=50, default="", blank=True)
    field_c = models.CharField("C", max_length=50, default="", blank=True)
  1. Make migrations makemigrations => 0003_animal_field_c.py
  2. Apply migrations migrate (0001,0002,0003)
  3. Squash migrations from 0002 to 0003 squashmigrations myapp 0002 0003 => 0002_remove_animal_field_b_squashed_0003_animal_field_c.py
  4. Add field D
class Animal(models.Model):
    field_a = models.CharField("A", max_length=50, default="", blank=True)
    field_c = models.CharField("C", max_length=50, default="", blank=True)
    field_d = models.CharField("D", max_length=50, default="", blank=True)
  1. Make migrations makemigrations => 0004_animal_field_d.py
  2. Apply migrations migrate (0004)
  3. Migrate backward migrate myapp 0003

Raise a KeyError on “field_b”

$ python manage.py migrate myapp 0003
Operations to perform:
  Target specific migration: 0003_animal_field_c, from myapp
Traceback (most recent call last):
  File "manage.py", line 22, in <module>
    main()
    ~~~~^^
  File "manage.py", line 18, in main
    execute_from_command_line(sys.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
    ~~~~~~~~~~~~~~~^^
  File "/venv/lib/python3.13/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/django/core/management/base.py", line 416, in run_from_argv
    self.execute(*args, **cmd_options)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
  File "/venv/lib/python3.13/site-packages/django/core/management/base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
  File "/venv/lib/python3.13/site-packages/django/core/management/commands/migrate.py", line 298, in handle
    pre_migrate_state = executor._create_project_state(with_applied_migrations=True)
  File "/venv/lib/python3.13/site-packages/django/db/migrations/executor.py", line 91, in _create_project_state
    migration.mutate_state(state, preserve=False)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/django/db/migrations/migration.py", line 91, in mutate_state
    operation.state_forwards(self.app_label, new_state)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/django/db/migrations/operations/fields.py", line 164, in state_forwards
    state.remove_field(app_label, self.model_name_lower, self.name)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/django/db/migrations/state.py", line 271, in remove_field
    old_field = model_state.fields.pop(name)
KeyError: 'field_b'

I’ve tried it against django 5.0.14, django 5.1.8, django 5.2, django git master and this particular commit which seems to incldue other fixes on backward bugs: Fixed #35595 -- Generated explicit RemoveIndex operations on model re… · charettes/django@d55d032 · GitHub

2

May be it is related to

3

Hello there,

I don’t think the issue you are experiencing has anything to do with there issues you linked, it appears to me to be a misuse of the migrate and squashmigration command.

You’ll notice that your 0004_animal_field_d migration depends on 0002_remove_animal_field_b_squashed_0003_animal_field_c and not 0003_animal_field_c which means that if you want to un-apply 0004_animal_field_d you should do migrate 0002_remove_animal_field_b_squashed_0003_animal_field_c and not migrate 0003.

In other worlds, once you’ve applied a migration that replaces others (0002_remove_animal_field_b_squashed_0003_animal_field_c replacing 0002_remove_animal_field_b and 0003_animal_field_c) the migration command should disallow you to unapply only parts of its constituents.

I believe the bug here is that migrate doesn’t prevent you from doing migrate 0003.

4

That sounds right to me @charettes. Perhaps we should allow specifically “roll back to 0003” though, replacing it to mean “back to the squash”?

I’d also to know what happens if 0004 does depend on 0003.

5

Oh thanks, I see.
Hmm.. in my real app, the error occurred when trying to migrate backward in another app. I will try to reproduce with a minimal sample.

6

I think that would be quite complex as that would require the following operation to happen

  1. Have migrate recognize that 0003 is replaced by the already applied 0002_remove_animal_field_b_squashed_0003_animal_field_c and choose to breakdown the latter into its units
  2. Concretely this means deleting the 0002_remove_animal_field_b_squashed_0003_animal_field_c record in django_migrations and then proceeding with the normal process of migrate 0003.
  3. Things get quite hairy if you have multiple migrations squashed migrations on top of each others as they have to be recursively broken down into atoms (migrations that don’t replace other migrations)

In practice the bug happens because migrate 0003 is interpreted as migrate 0002_remove_animal_field_b_squashed_0003_animal_field_c && migrate 0003 which results in some state alterations to be applied twice.

I’d also to know what happens if 0004 does depend on 0003.

I’m not sure I follow you here. How is that possible since 0004 is created after 0002 and 0003 are squashed into 0002_remove_animal_field_b_squashed_0003_animal_field_c which is the HEAD of the apps migrations at the makemigration time that results in 0004.

7

Thanks for taking time to analyze this issue.

You were true, I’ve tried to migrate backward to a migration already handled by a squashed one in first hand.
Trying to migrate backward on expected squashed migration is ok, even if there is over dependencies to squashed migrations on other apps.

It is a mistake from me. It may be useful to warn/stop user trying to do this kind of backward migrations.