Why have imports in `__init__.py` files instead of imports based on source code location?

[StephanieAG]

I was learning about Python packaging, so I was looking at Django as an example of a Python package and was confused re: Why isn’t there a Model.py file since we import django.db.models.Model when we make a model?

To elaborate:

I knew from using Django that you typically do the below to make a model:

from django.db import models

class ModelName(models.Model)
    ...

So when I was learning about Python packaging, I looked for django/db/models/Model.py, but it doesn’t exist:

So I explored and found Model in models/__init__.py:

# Imports that would create circular imports if sorted
from django.db.models.base import DEFERRED, Model  # isort:skip

which led me to django.db.models.base where I found the Model class:

class Model(AltersData, metaclass=ModelBase):
    ...

So that seems to answer the original question (the Model I was looking for is in base.py but imported in __init__.py for a reason related to avoiding circular imports which allows you to do models.Model without a Model.py file).

When I look at the rest of the models/__init__.py file though, it’s all imports except at the end where it defines __all__ as a list of all the imports in the file. And when I look at other __init__.py files throughout the django package, it looks like this pattern of importing other things into __init__.py files is commonly done.

But not all of the imports are under the “circular dependency” comment (most of them aren’t), and after thinking about this more, I’m thinking the comment actually isn’t explaining why Model is imported in __init__.py and instead is only explaining why those imports aren’t in the expected order for import statements.

Which brings me to my actual question: why does the django package import things in __init__.py files instead of having users import things from where they are located in the source code? E.g.
this (hypothetical):

from django.db.models.base import Model

class ModelName(Model)
    ...

instead of this (reality):

from django.db import models

class ModelName(models.Model)
    ...

For the record, I’m asking not because I have an opinion on this; I’m just trying to understand the reasoning behind putting the imports into __init__.py files – because I’m sure there’s a reason, and I just don’t understand it ( yet ).

[Lily-Foote]

Hi Stephanie!

Django makes it possible to import Model from django.db.models for the convenience of users of Django. We could achieve this by having a models.py file, but this would end up too big, so we’ve split this up into other files such as base.py to be more manageable for the contributors to Django itself.

As you noticed, the comment about circular imports is actually about using the tool isort to standardise the order of imports within __init__.py. Sorting imports is a useful convention for bigger Python projects like Django to help with readability. However, in this case if Django fully sorted imports this would cause a circular import problem, so we add an exception (# isort:skip) for DEFERRED and Model.

[KenWhitesell]

@StephanieAG

For the record, I’m asking not because I have an opinion on this; I’m just trying to understand the reasoning behind putting the imports into __init__.py files – because I’m sure there’s a reason, and I just don’t understand it ( yet ).

To add to the excellent answer provided above, I’ll point out that this is a technique available to you as well - and not just for models, either.

The benefits here include providing a degree of isolation for changes between modules. Reorganizing the code within the files in a module doesn’t need to affect code in other modules.

You could create your own model directory, place your models in a number of different files, and then import them into the model module in the __init__.py file.

Your other code using those models (e.g. views.py) wouldn’t need to change.

Using the same idea, you can split your views.py file into a number of different files inside a directory named views, where those are imported in the views/__init__.py file. Your urls.py file wouldn’t need to be changed when you reorganize those views. (It’s generally the responsibility of the person working on the views to ensure the __init__.py file is properly updated.)

[villager]

Ben Hoyt has a really good post, on designing package APIs. This quote explains the benefits of imports defined in __init__.py:

Recently in a library I maintain for work, we made a change to pull all the items defined in submodules up to the top level in __init__.py, making the class names significantly easier to discover. Instead of users having to guess, “Hmm, was ActiveStatus in the ops.charm or ops.model submodule?”, we now recommend simply using import ops and then ops.ActiveStatus. All the classes users need are right there.

Here’s the link to the module structure section. Django is actually used as an example of bad design.

Module and package structure - Designing Pythonic library APIs