Doc update: Describe steps for manually squashing migrations

Django Internals
1

Hi folks,

I’m looking to pick up the ticket on documenting the process for manually squashing migrations: as it looks like Mike doesn’t have the time to work on this. (Thanks Mike for putting in a valiant effort btw!) Also see existing thread on the mailing list.

I think we should push through something at least so as to let folks know that manually squashing is an easy option.

There are a few things that’d be good to clear up and get decisions on before submitting a PR:

  1. I think it’s worth continuing to document the squashmigrations command as the recommended “preferred” option. Carlton has mentioned they’ve used it regularly without any issue.

  2. The squashmigrations command can become problematic for larger projects. There are a few reports on issues with circular dependencies but I also had issues trying to get libs like django-pghistory/django-pgtrigger to work nicely with it. I think this is where we document “Here’s how you can manually squash migrations for when the squashmigrations command no longer works for you”.

  3. It seems there are 2 types of manual migration that people are doing:

a. “Resetting”: Deleting migrations, deleting database and simply rerunning makemigrations then applying fake migrations on the deployments as necessary.

b. “Replacing”: Moving migrations “out of the way”; running makemigrations with a distinct name; then moving the old migrations back and listing them in the “replaces” attribute similar to squashmigrations, while also copying any non-elidable operations (eg runpython/runsql) and their dependencies into the squashed migration. The process moving forward from here is the same for squashmigrations (wait til migrations are out of the “squash zone” then remove, update references, remove replaces, run migrate --prune, etc).

I’ve had success doing b. and found it quite easy to run through this particular project didn’t have circular cross-app dependencies. I’d recommend b over a as it makes it easier to manage deployments or other dev’s setup.

Is there anything anyone else would like to add? Would anyone strongly oppose b in favour of a? I heard some reports that running makemigrations from scratch with circular dependencies still won’t work? (I tested a small sample setup with very basic circular dependency and it seemed to run fine)

PS: There was a forum post a couple of years ago for an idea to add a --replace option which sounds similar to b albeit without the non-elidable detection. This would be useful for helping people run b.

Cheers,
David

2

Hey David.

That all sounds very sensible to me.

IIRC my objection to the previous PR was that it wasn’t very clear in the end — it introduced an extra notion of Trimming, which seemed to muddy the (already somewhat silty) water. I’d guess explaining clearly the options as they are today would be most of it (with maybe extra flags etc if needed from there).

Thanks for picking this up!

3

I took a first stab at this, however…

I was initially aiming to make the smallest update possible but realised that this section would be missing 2 useful new additions that have been introduced since this section was written:

  • makemigrations --update
  • optimizemigration

I really think we ought to include these under a general section on squashing/optimising (heading tbd?)

I was thinking:

h1 <heading tbd>

    h2 Optimising migrations
    
     - introduce the concept of optimising (will be referred to in sections below)
     - introduce the optimizemigration command
    
    h2 Updating migrations
     - introduce the makemigrations --update command and how it optimises
    
    h2 Squashing migrations
     - Explain the concept of squashing and how to coordinate deployments & other developers
    
        h3 Using squashmigrations
         - introduce the squashmigrations command (and possibly how it optimises)
            
        h3 Manually squashing
         - explain the steps involved if squashmigrations fails
1 Like
4

Draft PR up for feedback: Draft update to docs on squashing migrations by shangxiao · Pull Request #16843 · django/django · GitHub

1 Like
5

Hello there.
Recently I needed to manually squash the migrations, after trying to use squashmigrations and failing hard several times, I’ve used the a approach:

Here’s what I’ve done, hopefully this can help someone in the future.

Step 1

Remove all migrations from all apps and run makemigrations
Since I had quite a few apps, I wrote a small script to:

  • detect all apps with migrations and delete them;
  • recreate migrations

Just remember to take note of all the apps that had migrations recreated, you’ll need it on Step 4.

Delete all migration files inside the apps migrations folder Script 
import subprocess
from collections.abc import Generator
from pathlib import Path


def main():
    app_dirs_with_migrations = get_app_dirs_with_migrations()
    for migration_files in app_dirs_with_migrations.values():
        for migration_file in migration_files:
            migration_file.unlink()

    # Recreate all migrations
    subprocess.run(["uv", "run", "python", "manage.py", "makemigrations"], check=False)  # noqa: S607


def get_app_dirs_with_migrations() -> dict[Path, list[Path]]:
    project_root = Path.cwd().joinpath("your-root-project-dir")

    apps_with_migrations: dict[Path, list[Path]] = {}
    for app_dir in iter_path_dirs(project_root):
        migrations = migration_files_from_app(app_dir)
        if not migrations:
            continue
        apps_with_migrations[app_dir] = migrations
    return apps_with_migrations


def iter_path_dirs(path: Path) -> Generator[Path, None, None]:
    for sub_path in path.iterdir():
        if sub_path.is_dir() and sub_path.stem not in ("__pycache__", "static", "templates"):
            yield sub_path


def migration_files_from_app(app_dir: Path) -> list[Path]:
    migrations = []
    for app_sub_folder in iter_path_dirs(app_dir):
        if app_sub_folder.stem != "migrations":
            continue
        for file in app_sub_folder.iterdir():
            if file.stem.startswith("__"):
                continue
            migrations.append(file)
    return sorted(migrations)


if __name__ == "__main__":
    main()

Step 2

Running the new created migrations on a clean database
Since some of my models are using typing.Generic the migrations were using the typing.Generic class in the migrations.CreateModel operation, under the options bases, this was causing an error. So I needed to manually edit some of the files to remove the typing.Generic from the bases option inside some migrations.
To run the migrations from scratch on a clean database I’ve used docker for that.

test.yml 

services:
  django:
    build:
      context: .
      dockerfile: ./compose/local/django/Dockerfile
    image: project_local_django
    container_name: project_test
    depends_on:
      - test_postgres
      - redis
    environment:
      POSTGRES_HOST: test_postgres
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_PORT: 5432
      POSTGRES_DB: project
      REDIS_URL: "redis://redis:6379/0"
    command: pytest --cov=bid_capital_loan_service --cov-report=html:cov_report ${PYTEST_ARGS:-""}
    volumes:
      - .cov_report:/app/cov_report

  test_postgres:
    image: docker.io/postgres:16
    container_name: bid_test_postgres
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: project
    ports:
      - 5445:5432

  redis:
    image: docker.io/redis:6
    container_name: your_project_local_redis
    ports:
      - 6380:6379
bash script to run migrations 
docker compose -f test.yml down
docker compose -f test.yml up test_postgres -d
docker compose -f test.yml build
docker compose -f test.yml run django python manage.py migrate
docker compose -f test.yml down

Step 3

Running migrate --fake-initial on the target database

Since I had more than one environment, I’ve targeted the UAT environment for safety first (You need to make sure that all target databases are on the same version, all migrations that were deleted are applied on them).

uv run python manage.py migrate --fake-initial
This marked all the new “migrations” as being applied on the target database.

Step 4

Running migrate app_name --prune on the target database
After all the migrations were faked, I ran the migrate --prune command to remove all migrations that no longer existed. You need to run this for each app though, that’s quite painful.
uv run python manage.py migrate app_name --prune

Step 5

Assert that no migrations needs to be applied
Since we did some massive changes, it’s a good double check that you don’t have any migrations to be applied after the changes.
uv run python manage.py migrate --plan

Finale

That’s it, all apps now have one (or just a few) migrations. Notice that any local (other) environment will need to do steps 3-5 after this goes to your main branch (or just drop the database).