The Best Ways to Manage Django Database Migrations for a Big Project

Using Django
1

Hello everyone :hugs:,

I’m utilising Django (4.x) for a large-scale project right now, and I’ve run across a few issues with handling database migrations. Our PostgreSQL database is being used, and the project has significantly expanded in terms of data volume and code complexity. We’ve had to update the schema a lot over the years, and as the project grows, we’re now having problems with migration management.

The following are some particular problems I’ve been addressing:

Migration Conflicts: When we attempt to merge our branches, we frequently have migration conflicts because we have numerous developers working on the project. Exist any practical methods for lessening or preventing these disputes in a collaborative setting? :thinking:

Extended Migrations: Certain of our migrations, such as renaming big tables or adding non-nullable columns, are taking a long time. Downtime during production deployment worries me. Has anyone put any plans in place to reduce or eliminate downtime while performing large-scale database migrations? :thinking:

Migration Cleanup: Due to previous migrations, our migration folder is growing really large. Exists a suggested method for clearing out previous migrations without jeopardising database integrity? :thinking: Squashing migrations is something I’ve read about, but I’m not sure how safe or useful it is for a big project with constant updates.

Top Techniques: In general, I’m curious to learn about the best methods for overseeing migrations in big projects. What workflows, patterns, or tools would you suggest to improve the effectiveness and reduce the likelihood of errors in this process? :thinking:

Thank you in advance.

2

I think dumping the database to JSON and loading it into a new database might be a solution. Unfortunately, your project looks quite large.

If you try this solution:

Dump the database to JSON.
Edit the JSON for the new database fields.
Load it into the new database.
3

There are a few 3rd party packages that can help you address some of your problems, they do add a bit of friction, but in my experience it’s “good” frictions, because they help you discover issues earlier, before they are deployed and reach production.

  • django-linear-migrations forces you to have a linear migration graph and flags conflicting migrations as git conflicts, so you spot migrations conflicts before merging. This one is a must have, small overhead, but saves me from most issues.
  • django-pg-zero-downtime-migrations adds a few checks to avoid locks and make your migration fails if it takes too long, instead of your migration potentially taking your site down. You can configure it to raise errors if you write a migration which isn’t backwards compatible (e.g adding a new non-nullable field)
  • for squashing migrations, I wrote django-remake-migrations which is helping you do reset of all your migrations. It has a few rough edges, but is worth a try if you want to squash lots of migrations.

Hope that helps!

1 Like