Unexpected migration behaviour with django.contrib.auth

Hi there all,

I have a fairly common pattern I find myself repeating in an application I’m working on, where I need to define a new model class, then give some group of users CRUD permissions on it.

The way I do this is to create the model, and generate a migration for it, then, in another migration, to assign the correct Permission objects to the users I’m interested in giving them to. However, this leads to a slightly tricky, unexpected (at least, to me) behaviour which I’ll explain.

I’ve worked up an example in a gist so that you can follow along.

  1. I start by defining my new class (in the example, VerificationBasis)
  2. I then run manage.py makemigrations, which results in migration 001 in the example, which creates the table for the class itself.
  3. I then create a new empty migration, and add the code to assign the required permissions to the correct users (in the example, CRUD permissions on the new VerificationBasis class to my “admin” user group - you can see this in migration 002 in the example).
  4. I run manage.py migrate

..when I do this, both migrations run and report “OK”, the VerificationBasis table is created, along with the Permission objects autocreated by django.contrib.auth, but they don’t get assigned to the “admin” group - it’s as if the second migration never ran at all.

I later realised that if I run the migrations one at a time (running manage.py migrate immediately after makemigrations, then creating and running the second permissions migration, everything works correctly. However, in general, I can’t guarantee that the migrations will be run in this matter - for example when we deploy to production, our CI pipeline runs all pending migrations, and can’t know that it needs to run them one at a time.

After some digging, I realised the reason for this. django.contrib.auth automatically creates Permission objects for new models, but it does so in the “after migrate” callback, which runs after all migrations have terminated. Therefore, this explains the problem - when i run both migrations together, the table is created, the second migration is run (and is essentially a no-op because the permissions it is looking for don’t exist), then the django.contrib.auth callback is run to create the permissions.

At first glance, i figured this was a bug, but it seems like such a common use case that I must be doing something wrong. Therefore, i’d love to know if there are any patterns or best practices for doing these sorts of permissions changes in django apps, or how anyone else approaches it!

Many thanks,

Tim

Hello there!
I’ve also faced this issue, and after several attempts of setting-up permissions on migrations files, I’ve just given up. The easiest way I’ve found is to create a management command that setup my predefined groups permissions, and run it after the migrate command.
My usecase was that I have a separed set of groups that are related to a specific frontend, and everytime a new deploy is done the groups permissions are reset to their defaults (by running the management command), so I don’t change these groups on the admin, but on the source files.
If you want to take a look into the source code I would be happy to share (even though it’s a bit of a mess and has other usages beside setting up permissions).

Hi @timcowlishaw, looking at your gist I am a bit concerned regarding your setup-permissions migration for the following:

  • there are no imports for auth models (which would be bad) neither the use of historical models (which is the preferred way to do this)
  • there is no auth app in dependencies, thus if you are just running these migrations those models are not on the DB yet (or django is not aware of this). Without any dependency for a migration which creates Group and Permission models it is not possible to write to these tables.

How is your real setup? Does these migrations really run?

Depending on the auth models doesn’t do anything, because the problem that the user is facing is that the Permission objects (for any model) are only created after a “complete migrate” command is executed.

The point is, normally when you’re developing the first thing you do when start off a project is migrate the database, this creates the Permission objects for the installed apps.
After a while you add a new model, then run migrate. Then the Permission objects are created for that model.

The problem is when you add a new model, and before applying the migration, you try to setup permissions (by creating a RunPython operation or a new migration file with a RunPython operation) for that specific model on a Group per example, the permissions objects for that model won’t be available if both migrations are applied together.
The same problem shows up when you apply all migrations at once, per say, on CI to setup a test database, since no “complete migrate” command has been executed before these scripts that setups permissions will fail/raise because permissions objects will be missing on the database.

Hey folks,

Thanks so much for the replies! @leandrodesouzadev - I really like your strategy of having a separate set_permissions management command - I guess the idea is to make this idempotent, so that it can be run after every migrate whether or not any new permissions need to be created, and will always result in the correct permissions state? Thank you!

@sevdog , aha, yep, you’re quite correct - the gist is an “edited highlights” of the parts of my app that give the problem, the app itself is open source (and you can see the PR that prompted this question here, but there’s a lot going on there so figured i’d try and pull out a more focused example). Leandro is correct though - relying on django.contrib.auth isn’t needed in this instance, as it’s loaded in INSTALLED_APPS, and the permissions are getting created, just in a slightly unexpected order.

I now (thanks to the “related posts” feature) suspect this is the same underlying issue that is described here which also proposes another solution - explicitly calling django.contrib.auth’s permissions syncinc function from within each migration that changes the schema, but i think i favour Leandro’s solution, as it wouldn’t require us to remember the fix every time we change our schema, just to ensure that permissions are always set in the additional admin command to be run after all migrations.

Thanks all!

I will be sharing the solution here. It has a lot of “noise”, since i’m out of time right now. But I believe that you will be able to pick it up the idea.

Management command

# bid_capital_loan_service/users/management/commands/setup_bff_groups.py

from datetime import timedelta

from django.conf import settings
from django.core.cache import cache
from django.core.management.base import BaseCommand

from bid_capital_loan_service.users import roles

LAST_SETUP_VERSION_CACHE_KEY = "users:bff-role:last-setup-version"


class Command(BaseCommand):
    def handle(self, *args, **options):
        if not settings.DEBUG:
            return self._do_setup_on_production(*args, **options)
        return self._setup_groups()

    def _do_setup_on_production(self, *args, **options):
        current_version = settings.SYSTEM_RELEASE_VERSION

        last_setup_version = cache.get(LAST_SETUP_VERSION_CACHE_KEY)
        if current_version == last_setup_version:
            self.stdout.write(f"Skipping setup for {current_version=}, already setup before", self.style.WARNING)
            return
        self._setup_groups()
        cache.set(LAST_SETUP_VERSION_CACHE_KEY, current_version, timeout=timedelta(days=1).total_seconds())

    def _setup_groups(self):
        self.stdout.write("Setting-up BFF groups")
        roles.setup_frontend_groups()

The roles module
Here are the magic, doing some static type runtime behavior.
(Again there’s a lot of noise here, the method being called by the management command is setup_frontend_groups)

# bid_capital_loan_service/users/roles.py

from __future__ import annotations

from typing import TYPE_CHECKING
from typing import Annotated
from typing import Any
from typing import Literal
from typing import get_args

import structlog
from django.contrib.auth.models import Permission
from django.db.models import Case
from django.db.models import IntegerField
from django.db.models import Q
from django.db.models import When

if TYPE_CHECKING:
    from collections.abc import Generator

    from users.models import Group
    # this is the same from django.contrib.auth, but mypy thinks that it's not but cant import this at runtime here
else:
    from django.contrib.auth.models import Group


class PermissionCTCodename:
    """A list of permissions names that the group specifically have"""

    def __init__(self, *perms: str):
        for perm in perms:
            app_label, found, codename = perm.partition(".")
            if not all([found, app_label, codename]):
                raise ValueError(
                    r"Invalid permission name, must follow the format `{app_label}.{codename}`",
                    "Received: ",
                    repr(perm),
                )
        self.perms = set(perms)

    def __repr__(self):
        return f"{self.__class__.__name__}({self.perms})"


class InheritPermissionsFromGroups:
    """Define one or more groups that the group inherits permissions from. If the target inherited group itself inherits
    from another group, then permissions will be inherited recursively"""

    def __init__(self, *group_names: str):
        self.group_names = group_names

    def __repr__(self):
        return f"{self.__class__.__name__}({self.group_names})"


class FrontEndGroupPriority:
    """Priorities closer to 0 have higher precedence"""

    MAX_PRIORITY = 99

    def __init__(self, priority: int = MAX_PRIORITY):
        assert 0 <= priority <= self.MAX_PRIORITY, "Not within valid priorities"
        self.priority = priority

    def __repr__(self):
        return f"{self.__class__.__name__}({self.priority})"


FrontEndGroupDefinition = (
    Annotated[
        Literal["[Front] Operador (treinamento)"],
        PermissionCTCodename(
            "fgts_loan.simulate_fgts_loan",
            "fgts_loan.view_own_simulated_fgts_loan",
        ),
        FrontEndGroupPriority(90),
    ]
    | Annotated[
        Literal["[Front] Operador (Sem visão IA)"],
        PermissionCTCodename(
            "fgts_loan.submit_proposal",
            "fgts_loan.change_own_proposal",
            "users.view_own_operations",
            "bff.view_dashboards",
        ),
        InheritPermissionsFromGroups("[Front] Operador (treinamento)"),
        FrontEndGroupPriority(85),
    ]
    | Annotated[
        Literal["[Front] Operador"],
        PermissionCTCodename(
            "users.view_ai_operations",
        ),
        InheritPermissionsFromGroups("[Front] Operador (Sem visão IA)"),
        FrontEndGroupPriority(80),
    ]
    | Annotated[
        Literal["[Front] Operador (back-office)"],
        PermissionCTCodename(
            "fgts_loan.assign_proposal_to_another_user",
        ),
        InheritPermissionsFromGroups("[Front] Operador"),
        FrontEndGroupPriority(75),
    ]
    | Annotated[
        Literal["[Front] Supervisor"],
        PermissionCTCodename(
            "users.create_operator",
            "users.view_other_operators_operations",
            "fgts_loan.view_loansimulation",
            "fgts_loan.change_loanproposal",
            "fgts_loan.view_operational_report",
            "referrals.view_referrallink",
            "referrals.add_referrallink",
            "referrals.change_referrallink",
            "referrals.view_loanproposalreferral",
            "users.control_product_access",
        ),
        InheritPermissionsFromGroups("[Front] Operador"),
        FrontEndGroupPriority(70),
    ]
    | Annotated[
        Literal["[Front] Supervisor (back-office)"],
        PermissionCTCodename(
            "fgts_loan.assign_proposal_to_another_user",
        ),
        InheritPermissionsFromGroups("[Front] Supervisor"),
        FrontEndGroupPriority(65),
    ]
    | Annotated[
        Literal["[Front] Financeiro"],
        PermissionCTCodename(
            "comissions.view_resellercomissionpaymentprocesspayment",
            "comissions.view_resellercomissionpaymentprocess",
            "comissions.view_loanproposalcomission",
        ),
        InheritPermissionsFromGroups("[Front] Supervisor (back-office)"),
        FrontEndGroupPriority(60),
    ]
    | Annotated[
        Literal["[Front] Administrador"],
        PermissionCTCodename(
            "reseller.add_reseller",
            "reseller.change_reseller",
            "reseller.view_reseller",
            "reseller.add_resellerselfhiringdomain",
            "users.view_user",
            "users.change_user",
            "users.add_user",
            "bff.define_reseller_target_production",
            "bff.change_theme",
            "bff.add_theme",
        ),
        InheritPermissionsFromGroups("[Front] Financeiro"),
        FrontEndGroupPriority(0),
    ]
)


def _get_metadata_by_type[T](definition, t: type[T]) -> T:
    metadata = getattr(definition, "__metadata__", ())
    return next((meta for meta in metadata if isinstance(meta, t)), t())


def _get_group_definition_by_name(group_name: str):
    for group_def in get_args(FrontEndGroupDefinition):
        definition_name = get_args(get_args(group_def)[0])[0]
        if group_name == definition_name:
            return group_def
    raise ValueError("No group definition with name", group_name)


def _get_recursive_groups_permissions(groups: tuple[str, ...]):
    perms: set[str] = set()
    for group in groups:
        group_def = _get_group_definition_by_name(group)
        perms |= _get_metadata_by_type(group_def, PermissionCTCodename).perms
        inheritance = _get_metadata_by_type(group_def, InheritPermissionsFromGroups)
        perms |= _get_recursive_groups_permissions(inheritance.group_names)
    return perms


def _get_group_names_with_recursive_perms() -> Generator[tuple[str, set[str]]]:
    for group_name, definition in _get_group_names_definitions():
        perms = _get_metadata_by_type(definition, t=PermissionCTCodename).perms
        assert len(perms), "Expected at least one Permission to be defined on the group"

        inherited_groups = _get_metadata_by_type(definition, InheritPermissionsFromGroups)
        perms |= _get_recursive_groups_permissions(inherited_groups.group_names)
        yield group_name, perms


def _get_group_names_definitions() -> Generator[tuple[FrontEndGroupDefinition, Any]]:
    for group_def in get_args(FrontEndGroupDefinition):
        literal_type = get_args(group_def)[0]
        yield get_args(literal_type)[0], group_def


def setup_frontend_groups():
    """Setup all groups creating or updating them with the required permissions"""
    for group_name, _ in _get_group_names_definitions():
        setup_group(group_name=group_name)


_logger: structlog.typing.FilteringBoundLogger = structlog.get_logger(__name__)


def setup_group(group_name: FrontEndGroupDefinition) -> Group:
    """Create or update a group with the defined permissions. Can be called many times for the same group with no
    side-effects. It won't change custom permissions added manually (e.g through the admin), it only adds the defined
    permissions to the group permissions m2m."""
    logger = _logger.new(group_name=group_name)
    logger.debug("Setting-up group")

    perms = next((p for n, p in _get_group_names_with_recursive_perms() if n == group_name))
    logger.debug("Got perms", perms=perms)

    group, created = Group.objects.get_or_create(name=group_name)
    if not created:
        logger.debug(f"Group {group_name} already existed")

    perm_filter = Q()
    for perm in perms:
        app_label, codename = perm.split(".")
        perm_filter |= Q(content_type__app_label=app_label, codename=codename)
    permissions = Permission.objects.filter(perm_filter).select_related("content_type")
    db_perms = {f"{p.content_type.app_label}.{p.codename}" for p in permissions}

    missing_perms = perms.difference(db_perms)
    if missing_perms:
        logger.warning("One or more permissions were not found", missing_perms=missing_perms)
        raise ValueError("Missing permissions")
    group.permissions.add(*permissions)
    return group

Aha, that’s fantastic, thanks so much @leandrodesouzadev!

I hope you have solved the issue for now but the way i did was to setup postmigration signals on my dependant app’s models, for example in my case i have created a roles table in which it have permissions id i would manulaay print them and them pass id or codenames eg, view_cusotmers, add_customers , i prinited those id and seed those roles inmy post migrations signals . That’s how i solved that , unfortunatly I didn’t have to post the question and wait for answers