Check for ghost migrations (opposite of "You have N unapplied migration(s).")

Django Internals
1

Hey !

With various management commands (runserver, migrate --check, etc.), Django lets me know when I have unapplied migrations, as obviously this will lead to undefined behaviour.

However, Django says nothing for the opposite, where you applied a migration to the database, but then changed to a version of the app that doesn’t need this migration. I think this is just as bad in terms of indeterminism. In my practice, I often run into this issue when developping (switching branches…), and I can see how this could also happen in prod (after rolling back to a previous version, etc).

Googling it, I didn’t find much on this topic, except that South (but I came to Django after that) had such a warning for “ghost migrations”.

Now the questions:

  • am I the only one often stung by this ?
  • do you people have a strategy for dealing with it ?
  • are you aware of a reason this wasn’t ported to Django when South was integrated ?
  • wouldn’t it make to implement this in core (it would run at the same time than the missing migrations check) ? or better as a package ? (or not at all ?)

I didn’t look at it yet, but conceptually it seems relatively easy to implement: for each installed app, check if there’s a migration listed in the migrations table that doesn’t exist in the code. Of course that wouldn’t work for uninstalled apps (but I think that’s not really solvable), and also squashed migrations should be taked into account which complexifies a bit the check, but all in all sounds like something doable.

Thanks !!

Olivier

2 Likes
2

Does migrate --prune address your use case?

3

<opinion>
This would be of no value to me. There are too many holes in what can happen here to trust such a tool.

Yes, I have been bitten by things like this, which is why I have very strict practices regarding the development and uses of migrations in branches.

Briefly, migrations can be developed on a branch, but those migrations do not get committed. As the last step before a branch gets merged, the new migrations are removed, the current main branch gets merged into the development branch, and makemigrations is run at that point. It is then tested with the current production schema to ensure that the migrations work. If everything tests out ok, then the branch can be merged into main.

Yes, it adds more work, but it works for me and prevents many of the types of situations you describe.

Regarding the need to rollback a production release, I always take a backup of the database before deployment. That’s the only way I can ensure getting back to a functional version.
</opinion>

That doesn’t address the possibility of a migration being changed (such as in the case of a data migration), or even the situation where the same model is modified in a different way, such that the name of the migration ends up being the same.

It also doesn’t address the potential need to reverse migrations that can’t be reversed, such as the removal of a field.

From my perspective, the only way to prevent problems with “ghost” migrations is to prevent them from ever happening.

4

This issue does come up when switching branches or rolling back versions, and it can lead to some tricky situations. One strategy I use is to keep track of the migrations applied manually through version control or a script that checks for orphaned migrations. I agree that implementing a check for “ghost migrations” in Django could be useful. A possible approach could involve comparing the migrations listed in the database against the actual codebase, as you mentioned. This could be part of the core or as a package, but it would definitely help prevent indeterminism.

5

I’ve felt for a while that we should have the ability to determine when a migration was applied in a state that’s different than what it was originally defined as. The concept of migrations is really hard to grasp. I think devs would benefit from being told, “hey, you changed this, are you sure that’s what you wanted?” more often. I remember having a conversation (maybe with @charettes) on how this might be implemented, but I could be imagining things.

6

I think a small solution here would be nice. At times I’ve had to alert colleagues that I’ve rewritten the migrations on a shared branch and that they should reverse migrate a bit before pulling the latest. Some extra reinforcement from the terminal would help make this obvious.

Some guesses re: why this hasn’t been implemented yet:

  • uncertainty about what follow-up action to recommend.
  • lack of the migrate --prune command until recently

Something like this wording might express enough uncertainty?

The following migrations are recorded as applied but no longer exist in your project:
…
Your project may not work correctly if they are reversible and are intended to be reversed.
To reverse the migrations, recover them (e.g. by switching branches) and run:
“manage.py migrate <app_name> <last_migration_on_disk>”
If you are certain the migrations do not need to be reversed, run:
“manage.py migrate <app_name> --prune” to remove references to them.