how to avoid circular imports in helpers.py

Using Django Mystery Errors
1

As the title indicates, my issue is with circular imports in the helpers.py from one of my apps. Here’s the directory structure and some code:

/gameskedge
    /apps
        /users
        /teams
        /web
        /westmarches (has _init_.py and empty models.py
            /games (has __init__.py and models )
                /helpers.py (this is where I have the issue)
            /profiles (has __init__.py and profile models)
            /schedules (has __init__.py and models )
    /gameskedge (project folder with settings.py)

here’s some code:

profiles/models.py

from django.db import models
from gameskedge import settings
from django.core.validators import MaxValueValidator, MinValueValidator
from apps.teams.models import BaseTeamModel


class Profile(BaseTeamModel):
    """Profiles link Characters to Teams Users (users.CustomUser Class) through Rosters"""
    player = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT)
    name = models.CharField(max_length=32, default=player, blank=True, null=True)
    is_active = models.BooleanField(default=False, null=False, blank=False)
    is_default = models.BooleanField(default=False, null=False, blank=False)
    gamesessions = models.ForeignKey('westmarches_games.GameSession', on_delete=models.PROTECT, null=True, blank=True)

    def __str__(self):
        return f'{self.name} [{self.player}]'

...

games/models.py

import uuid

from django.db import models
from apps.teams.models import BaseTeamModel
from apps.westmarches.profiles.models import Profile


class GameSession(BaseTeamModel):
    """A Game Session with players"""

    GAME_SESSION_STATUS = [
        ('new', 'New'),
        ('open', 'Open'),
        ('closed', 'Closed'),
        ('full', 'Full'),
        ('inprogress', 'In progress'),
        ('completed', 'Completed'),
        ('cancelled', 'Cancelled')
    ]

    GAME_REQUEST_STATUS = [
        ('created', 'Created'),
        ('requested', 'Requested'),
        ('accepted', 'Accepted'),
        ('rejected', 'Rejected')
    ]

    session_status = models.CharField(max_length=10, null=True, blank=False, choices=GAME_SESSION_STATUS, default='new')
    request_status = models.CharField(max_length=9, null=True, blank=False, choices=GAME_REQUEST_STATUS, default='created')
    start_datetime = models.DateTimeField(default=None, blank=False, null=False)
    end_time = models.TimeField(default=None, blank=False, null=False)
    requester = models.ForeignKey(Profile, related_name='requests', on_delete=models.CASCADE, default=None, blank=True, null=True)
    officiant = models.ForeignKey(Profile, on_delete=models.CASCADE, default=None, blank=True, null=True)
    members = models.ManyToManyField(Profile, related_name='profiles', through='GameGroup', blank=True)

    def __str__(self):
        return f'{self.start_datetime}'

...

class GameGroup(BaseTeamModel):
    "A set of profiles invited to a GameSession"

    game_session = models.ForeignKey(GameSession, on_delete=models.CASCADE)
    profile = models.ForeignKey(Profile, on_delete=models.CASCADE)


class GameInvite(BaseTeamModel):
    """An invitation to a GameSession"""

    INVITE_STATUS = [
        ('composed', 'Composed'),
        ('delivered', 'Delivered'),
        ('received', 'Received'),
        ('accepted', 'Accepted'),
        ('rejected', 'Rejected'),
    ]

    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable = False)
    status = models.CharField(max_length=11, null=True, blank=False, choices=INVITE_STATUS, default='composed')
    sender = models.ForeignKey(Profile, on_delete=models.CASCADE, null=True, blank=True)
    game_session = models.ForeignKey(GameSession, on_delete=models.PROTECT, related_name='game_invites')
    recipient = models.ForeignKey(Profile, on_delete=models.CASCADE, related_name="received_invites", null=True, blank=True)
    received_date = models.DateTimeField(default=None, blank=False, null=False)
    response_date = models.DateTimeField(default=None, blank=False, null=False)

    def __str__(self):
        return f'{self.game_session}:'

After I added this code, I started to get the circular import issues:
games/helpers.py

from .models import GameInvite, GameSession
from apps.westmarches.profiles.models import Profile


def create_game_session(self, start_datetime, end_time, requester, officiant):
        game_session = GameSession.objects.create(start_datetime, end_time, requester, officiant)
        game_invite = self.create_game_invite(game_session, requester, officiant)
        self.send_invitation(game_invite)
        return game_session

def create_game_invite(game_session: GameSession, recipient: Profile, sender: Profile=None):
    actual_sender = sender or game_session.requester 
    new_game_invite = GameInvite.objects.create(
        status='composed',
        sender=actual_sender, 
        game_session=game_session,
        recipient=recipient
    )
    game_session.game_invites.add(new_game_invite)
    game_session.save()
    return new_game_invite

def process_game_invites(game_session: GameSession):
    for game_invite in get_unsent_invites(game_session):
        send_invitation(game_invite)

def send_invitation(game_invite: GameInvite, recipient: Profile=None):
    receiver = recipient or Profile.objects.get(pk=game_invite.recipient.pk)
    receiver.game_invites.add(game_invite)
    receiver.save()
    game_invite.status = 'delivered'
    game_invite.save()

def get_unsent_invites(game_session: GameSession):
    return GameSession.objects.get(pk=game_session.pk).filter(status='composed')

It’s clear that the top level imports in helpers.py are the issue. At the top of games/models.py we have the import for apps.westmarches.profiles.models.Profile, so when I import Profile as well as GameSession and GameInvite in the helpers, it’s obviously the cause.

I know I can import the modules needed on a per function bases under helpers.py, or that I can remove those imports on the models themselves and replace my orm definitions like fk’s and m2m’s, etc… using the module and class names as single quoted strings, but both of those feel like workarounds that are missing a larger more important point about project organization.

What are my options for refactoring or reorganizing this code to address the issue?

One thought was that I should have a core app somewhere that contains all the helpers, and it would make sense for that to be under gameskedge/apps/westmarches/core but that won’t do anything to alleviate the circular import issue. I’ll still have it, just coming from core/helpers.py instead of games/helpers.py

At the moment, these helpers are all focused entirely on implementing business logic, so I think helpers.py or utils.py is the right place for them. But the actual business logic they’re providing is tied to creating a GameSession and updating/sending invitations to other players. Based on what they’re doing, maybe they’re not helpers, and I’m missing something in terms or best practices/favored patterns for creating and configuring model objects after initial creation.

Any help is welcome at this point. I’m mainly looking to understand what I may be missing in terms of a better way to approach this as well as heading off design decisions that may become impactful at a point down the road when fixing them requires a lot more effort.

2

I actually just realized that I got around this when working through similar issues between profiles and games, as seen line 13 from profiles/models.py:

gamesessions = models.ForeignKey('westmarches_games.GameSession', on_delete=models.PROTECT, null=True, blank=True)

I forgot that I had done this to get around the same problem when I first encountered it between profile and game apps. I assume anyone reading will spot that and wonder why it’s OK there but not in games or helpers.py. The answer is that I didn’t like it then either, I just felt like I needed to keep making progress so I used what works. Now I’m at the point where I want to try and address it across the board by reorganizing or refactoring with a pattern that I can follow to avoid having this come up repeatedly. I’ve been counciled that it’s not such a big deal to use the string represenation in the model relationship definitions or to import the needed functions within helpers.py. It just feels like a hacky workaround to me, and one that will make my life harder in the future when this code needs to be maintained

3

Ok, so there are a couple points here worth making.

First:

Not only is it “not a big deal”, but it’s documented as existing for this specific reason. That hardly makes it a “hacky workaround”.

The other, more opinionated topic is the very existance of this “helpers.py” file. Even though you have chosen a different name, you’re effectively talking about something usually referred to as a “services” layer.

It’s interesting to me that we’ve had at least three separate discussion threads on this basic topic in the past week or so.

I suggest you see the threads (and the links contained within them) at:

(among others)

Reader’s Digest version: If I were coding this, there wouldn’t be a “helper” file. As far as I’m concerned, the logic belongs elsewhere - either the views.py or the models.py as appropriate.

4

I think where I’m not connecting dots on it not being a big deal is that it’s harder to keep track of the links between models in different apps when the imports aren’t for the whole file, and you need to make more changes.

Thanks for singling out those two links. They were in my search results and the set was large so it’s good to know which to focus on.

I’m having trouble applying the readers digest version to my specific example, but at this point I guess I don’t know what I don’t know and need a closer look at those threads.

5

<opinion>
These are the types of issues you end up creating for yourself by trying to make files “too discrete”.

There are other threads around here about when you should be creating separate apps, or otherwise adopting a file / directory structure that doesn’t match the Django defaults, and I consistently express the opinions of “One app until you can’t”, and "Don’t create a non-standard directory structure unless you’re really sure you need to do this and understand the implications.

For some other thought on this see Why do we need apps?, Project organization- when is something a separate app?, and Is the structure of the project correct?

</opinion>

6

Here’s where I think I need to go next:

from django docs, this bit

This sort of reference, called a lazy relationship, can be useful when resolving circular import dependencies between two applications.

tells me to just use it and move on. I am still not sure on how to answer the next question, which is “where do useful functions for messing with the models go?”

I think the answer is that the view handles pulling together the models from across the apps, so make sure that the individual model and their managers give the views everything they need. So if I know that new gamesession always needs a new gameinvite, the forms in the views are going to be where I ask the orm to give me those things. I think that means that I need to be looking at managers for my models?

While typing this I see that you posted about dropping the separation between apps. I guess something that’s catching me up there is that I thought django ‘expected’ apps to be closely tied to a small set of model functions. I guess I’m thinking too small, based on some of the posts I see where you discuss your models.

It seems like instead of separating my players from their games, and their games from their invites, the fact that I’m always going to have all three in the same place means that they belong in the same app? I think the first conclusion about the models and managers stands, even when combining the apps?

7

<opinion>
They are an important component in a system - one tool among many.
</opinion>

In general, yes, that would be my conclusion.

I know in one of my posts somewhere (can’t find the link off-hand) I mention that there are only two situations where we consider separating a project into more than one app.

  • When there’s a self-contained bit of functionality that we envision wanting to use across multiple projects. For example, we have a module that can decode a certain type of data packet. We use that code in a lot of places, making it a perfect candidate for being a separate app. We’ve also packaged it as a module for internal use across project.
  • When the project is so large that we find it impractical to maintain it as a single app. That’s the situation that we knew we were going to encounter with our “flagship” project that you’ve seen me reference elsewhere.

Since that time, we’ve now encountered a third situation that also qualifies, and that’s when you have truly independant functionality that executes in a different runtime environment. Celery tasks and Channels worker processes are the two main examples of this.

So, to circle back around to your first question in this post:

Our answer is “all of the above”. Some in views, some in models, and when appropriate, some in processes for other apps. Side note: There’s nothing that says that only view functions or classes belongs in views.py. That file is a perfectly suitable location for functional code that might be shared among multiple views.

8

Thanks @KenWhitesell this has helped me make some important decisions about structuring the project and app and clarified why/how my thinking was leaning towards a service layer even though I didn’t realize it or think of it that way at first.

I have a some follow up questions on the topic.

  1. What’s the thinking on multiple model files within an app for logical separation, ease of identifying and grouping related code visually? I get the feeling that this is still “too discrete” because I’ll need to reference the other files and do lots of imports within the functions that need them.

  2. If I just use one model file, I should expect to define the orm relationships with the string references to the models, right? Because I will inevitably have references to things that are defined further down in the file? IMO, it makes sense to just make it part of my style definition for the project to always uses the string references even if it makes my life a little harder when refactoring. sound reasonable?

  3. I know I will have async tasks to manage via celery. I think it makes sense to keep the async runner in a separate app, so I’ll probably plan for that to be a separate app, thinking along the lines of your third case above. That seems like a fine choice and easy to change if I decide it’s overkill. Not a question, but germane to the actual question: I know that the next thing I’m adding after I get the game scheduling and invitation/messaging stuff worked out will be a CMS. I’m looking at wagtail for that. Is a package like that which is large on it’s own, something that makes sense to plan as a separate app as well? My instincts say no, because, again, these things are highly coupled. A wagtail blog or wiki post is always going to be tied to a user and a game session.

Again, thanks for the time and guidance

9

Some people like doing it, we don’t. It really is an issue of personal taste. We find it easier to find code in one file rather than having to search across multiple files. Tools like the “Outline” feature in VSCode makes it easier to find classes in a file.

We tend to organize the models such that there aren’t forward references. No, it’s not always desirable to do it this way, but fortunately those situations tend to be rare.

That’s a personal choice - I don’t see an issue with it either way.

However, I think you’ll find, that if you do start with a very conservative organizational structure, you may find you’re not going to end up doing as much refactoring as you may think.

That depends upon the data that the tasks are working with. If those tasks are responsible for working with your primary app models, it may make sense to keep the tasks as part of your main app.

Wagtail is an app. (See Integrating Wagtail into a Django project — Wagtail Documentation 6.0.2 documentation) Beyond that, I can’t advise you. (I’ve never really worked with Wagtail, aside from some very brief experimentation.)

On a different note, and one that is more an issue of perspective than anything else, is that I don’t consider a reference to data as being a “coupling” in the sense that the term is generally used.

Changing topics slightly, I guess there’s a mindset or approach that we use as a general principle that really underlies everything that I’ve described as how we do things, and that’s to not-overthink this. Don’t try to preemptively address concerns or difficulties that only may occur, as opposed to those that are occurring. You can spend a lot of time and effort trying to create the perfect architecture, only to find out later that 90% of what you’ve tried to do hasn’t created any demonstrable benefit.