Using default permissions in migrations and fixtures for tests

We are having a problem with providing data of auth models Permission and Group for running tests. In particular we need a way to construct test cases akin to “Can user X as part of group Y access this view/endpoint?”.

One way to do this, is to construct the required groups, assign them permissions and assign groups to users in the setUpClass method of a test case:

@classmethod
def setUpClass(cls):
    super().setUpClass()

    group = Group.objects.create(name="staff_users")
    group.permissions.add(Permission.objects.get(codename="add_logentry"))
    group.save()

    staff_user = CustomUser.objects.get(username="staff_user")
    staff_user.groups.add(group)
    staff_user.save()

However this seems cumbersome when Django provides the option of fixtures and data migrations. The latter would also be our preferred way to provide a number of default user groups for users running instances of our code base. We learned that it is not possible to provide objects of classes auth.Group, django.ContentType and auth.Permission in a fixture, although I’m not entirely sure why. All I have is a note to myself that says " very complicated interactions between QuerySet instances and fixture loading", which I think means that I wasn’t able to fully figure out the issue.

That leaves the option of a data migration, for example:

group_class = apps.get_model('auth', 'Group')
perm_class = apps.get_model('auth', 'Permission')

perms = defaultdict(lambda: {})
for perm in perm_class.objects.all():
    app_name = perm.content_type.app_label
    perms[app_name][perm.codename] = perm

for group_name, perm_defs in GROUPS.items():
    try:
        existing = group_class.objects.get(name=group_name)
        group = existing
    except group_class.DoesNotExist:
        group = group_class(name=group_name)
        group.save()

    for add_perm in perm_defs["to_add"]:
        perm = perms[add_perm[0]][add_perm[1]]
        group.permissions.add(perm)

where GROUPS defines which groups to add and which permissions they want. The permissions both include custom permissions and the four “add, change, delete, view” default permissions that are being automatically created for models. This migration worked fine when I wrote it, but it breaks running tests, which will fail with KeyError: 'add_logentry' while running migrations on the newly created test database, because no permissions exist yet when the code above tries to load them (checked it in a debugger). I made sure that the data migration has dependencies on all migrations that create models (and thus their permissions), but it still fails at that point.

I think what is happening here, which is also described in the documentation on default permissions, is that the permissions are not being created by a migration itself, but by the migrate process as a whole after all migrations have been performed. During testing all migrations are being performed in one call to migrate, so the permissions can’t be expected to exist at any point in any migration.

Long story short: Are we missing something or is the first option (with TestCase.setUpClass) the intended way to provide test data for the auth models?

Do you remember if you tried creating the fixture with natural primary and foreign keys via dumpdata?

No, all fixtures were created from scratch. To test this I just tried to put a Group into a fixture and… it worked. Now I look like a fool, because I don’t think we updated Django since the last time I tried and wrote that note. I do remember becoming very frustrated with the issue and writing it off as not possible, but I guess I just wasn’t thorough enough in getting to the root of the problem at the time.

Well, I suppose this solves the problem in a way. For tests we can now put the user groups in a fixture and require it in test cases. For providing clients with default user groups we can still use the fixture to load definitions, but will have to find a way to put that somewhere that isn’t a migration. That seems like a separate problem with a number of solutions readily available.