Hey !
With various management commands (runserver, migrate --check, etc.), Django lets me know when I have unapplied migrations, as obviously this will lead to undefined behaviour.
However, Django says nothing for the opposite, where you applied a migration to the database, but then changed to a version of the app that doesn’t need this migration. I think this is just as bad in terms of indeterminism. In my practice, I often run into this issue when developping (switching branches…), and I can see how this could also happen in prod (after rolling back to a previous version, etc).
Googling it, I didn’t find much on this topic, except that South (but I came to Django after that) had such a warning for “ghost migrations”.
Now the questions:
- am I the only one often stung by this ?
- do you people have a strategy for dealing with it ?
- are you aware of a reason this wasn’t ported to Django when South was integrated ?
- wouldn’t it make to implement this in core (it would run at the same time than the missing migrations check) ? or better as a package ? (or not at all ?)
I didn’t look at it yet, but conceptually it seems relatively easy to implement: for each installed app, check if there’s a migration listed in the migrations table that doesn’t exist in the code. Of course that wouldn’t work for uninstalled apps (but I think that’s not really solvable), and also squashed migrations should be taked into account which complexifies a bit the check, but all in all sounds like something doable.
Thanks !!
Olivier
2 Likes
Does migrate --prune address your use case?
<opinion>
This would be of no value to me. There are too many holes in what can happen here to trust such a tool.
Yes, I have been bitten by things like this, which is why I have very strict practices regarding the development and uses of migrations in branches.
Briefly, migrations can be developed on a branch, but those migrations do not get committed. As the last step before a branch gets merged, the new migrations are removed, the current main branch gets merged into the development branch, and makemigrations is run at that point. It is then tested with the current production schema to ensure that the migrations work. If everything tests out ok, then the branch can be merged into main.
Yes, it adds more work, but it works for me and prevents many of the types of situations you describe.
Regarding the need to rollback a production release, I always take a backup of the database before deployment. That’s the only way I can ensure getting back to a functional version.
</opinion>
That doesn’t address the possibility of a migration being changed (such as in the case of a data migration), or even the situation where the same model is modified in a different way, such that the name of the migration ends up being the same.
It also doesn’t address the potential need to reverse migrations that can’t be reversed, such as the removal of a field.
From my perspective, the only way to prevent problems with “ghost” migrations is to prevent them from ever happening.
This issue does come up when switching branches or rolling back versions, and it can lead to some tricky situations. One strategy I use is to keep track of the migrations applied manually through version control or a script that checks for orphaned migrations. I agree that implementing a check for “ghost migrations” in Django could be useful. A possible approach could involve comparing the migrations listed in the database against the actual codebase, as you mentioned. This could be part of the core or as a package, but it would definitely help prevent indeterminism.
I’ve felt for a while that we should have the ability to determine when a migration was applied in a state that’s different than what it was originally defined as. The concept of migrations is really hard to grasp. I think devs would benefit from being told, “hey, you changed this, are you sure that’s what you wanted?” more often. I remember having a conversation (maybe with @charettes) on how this might be implemented, but I could be imagining things.
I think a small solution here would be nice. At times I’ve had to alert colleagues that I’ve rewritten the migrations on a shared branch and that they should reverse migrate a bit before pulling the latest. Some extra reinforcement from the terminal would help make this obvious.
Some guesses re: why this hasn’t been implemented yet:
- uncertainty about what follow-up action to recommend.
- lack of the
migrate --prune command until recently
Something like this wording might express enough uncertainty?
The following migrations are recorded as applied but no longer exist in your project:
…
Your project may not work correctly if they are reversible and are intended to be reversed.
To reverse the migrations, recover them (e.g. by switching branches) and run:
“manage.py migrate <app_name> <last_migration_on_disk>”
If you are certain the migrations do not need to be reversed, run:
“manage.py migrate <app_name> --prune” to remove references to them.
Hello! I’m following up on the ticket that @jacobtylerwalls kindly opened to track this discussion. I’ve closed it as wontfix for now, but we can certainly revisit that decision as the conversation develops.
From my own experience, and in workflows where development branches are regularly rebased onto the latest main, ghost migrations typically appear in two scenarios:
- When migration reversals (especially custom ones) haven’t been fully considered. In those cases, reversing applied migrations before switching branches can help avoid inconsistencies.
- When rollback procedures in production don’t include reversing migrations (though ideally they would). On that note, I’m curious how a warning in
runserver, which isn’t used in production, would help in that particular case.
That said, I’m aware workflows can vary widely, and I’d like to better understand other contexts where this feature would be especially helpful.
If folks here have tried --prune, I’d be really interested to hear how well it worked (or why it didn’t). Maybe what’s missing is clearer documentation, or even a --dry-run flag for it?
@andrewgodwin Hi! I know you are busy this week, but if you have some time at some point, would you be able to share some rationale around this? Thank you!
It has been over a decade, so I’m afraid I can no longer remember the reasoning for omitting ghost migration checks when I wrote Django’s migrations.
My educated guess would be the presence of “squashed” migrations - where you can have migrations in the DB that are no longer on disk as they have been squashed - and I simply did not have time or was not able to write a check that took that into account.
--prune works, but you have to discover the need to use it, and you need to know that using it is appropriate (the migrations in question should be considered abandoned). Adding a dry-run or plan mode would address the second part, giving you a way to see at glance if things can be safely pruned.
Here is the separate ticket for adding dry-run/plan mode to --prune. Maybe we should evaluate these feature requests in tandem.
Re: squashed migrations, --prune fails gracefully when a replaced migration has not been fully transitioned to a normal migration.
On that note, I’m curious how a warning in runserver , which isn’t used in production, would help in that particular case.
Ah good catch, I should not have been so specific in the ticket title. I’m assuming we would hook into requires_migrations_checks, which is currently primarily leveraged by runserver but also createsuperuser and createpassword. I’m surprised we don’t use it in check, which is a good candidate for having in a deployment pipeline.
Pardon the double post, just saw the Discord comments. Simon mentioned the inability to point to the origin of the ghost migration as a drawback. In practice, after a one-time prune for really old ghost migrations that are clearly abandoned, this warning would appear immediately after switching branches, so it would be self-evident where the migrations were.
I agree with Simon and Natalia’s comments on Discord that ghost migrations in and of themselves are not a problem. They could be symptom of a problem depending on their content, but the same could be said for the current warning for the forward direction, which I frequently ignore when I know the migration is an SQL noop. (As of 5.1 makemigrations communicates this information with symbols, so we could reuse it in this warning.) I get it if this is too much feedback for a slim use case.