Modularization of Django

Django Internals
1

Firstly a quick disclaimer, the title isn’t quite right, but it’s the best I can think of right now.

I would like to get a community discussion around the potential solutions for including extra packages into an installation of a Django project without the code necessarily being included into core django codebase.

This topic has been discussed in the margins of other topics (namely the current GSoC project - keyboard shortcuts and django tasks db backend), but has yet to be firmly considered in of itself.

The Steering Council have been discussing this in relation to the 2 projects I just mentioned, and Carlton touched upon it in his DjangoCon Europe talk (and corresponding Stack Report).

Before we jump on possible solutions it’s worth agreeing on what the immediate and future goals.

As I see it the main goals are as follows:

  1. Allow for code (Javascript) to be shipped that doesn’t require the Fellows to maintain as their expertise is Python
  2. Widen the queue of items that can be shipped while maintaining quality & stability.
  3. Long term potentially allow for quicker iterations of certain features of Django if a developer wanted to opt-in to this behaviour. (This might be too extreme to discuss now!)

If I have any of the above wrong (which is likely) then please correct me. As I previously said if we could get the goals defined we would then be in a better place to discuss possible solutions.

2

I think the tasks db backend is a great place to begin this conversation. We’re looking to merge Jake’s PR for the Interface but there’s going to zero working backend in core as part of that. Deliberately so: Absolutely guaranteed the DB backend
is going to need rapid iteration, and we don’t want to tie this to Django’s release process until it’s settled.

But then we need to call out the requisite pip install django-tasks so that folks can actually give it a run.

I think this is fine.

It’s very much the way it should be. Eventually we’ll merge the task DB backend into core, once the obvious as soon as you run it in numbers issues have been ironed out.

It’s the Django way. This isn’t any different from say django-testdata which was merged after an initial period as a third party package, or as we’re hoping to do with template-partials
this summer. (And a million other examples.)

Equally the keyboard shortcuts — which is again a desired addition — should be put in a third package so people can test it — and let’s even promote it, so that we get wider adoption.

None of that really touches on modularisation, per se. It’s just how we give things a try, rather than experimenting in Django itself.

I suspect after an initial period of flux, the keyboard shortcuts will end up as a small amount of JS that we probably can just merge.

BUT let us imagine it ends up being a big chunk of JS that we don’t want in core, because it’s more maintenance than we want to take on… What then…?

Well, we make sure the right hooks are in Django, and then it continues to live as an external package, like DRF, or django-filter, or Wagtail.

The new ecosystem page is still a bit first-pass, which is OK: it is the first pass, but we should evolve that, and it can give guidance to packages we want to feature. Etc.

Again, though, this is just best practice of what we do already.

Your point was different:

For THIS I’m playing with a proof-of-concept for an opinionated package that installs a set of extras alongside Django in a single step. This isn’t something I have ready to go yet, but it’s coming — and I think others should play with similar. I think it’s the direction we need to look at. It’s what I called Battery Packs in my DjangoCon talk you mentioned above.

One starting thing I’d like us to look at is possibly adding pip extras for the existing additional installs I need… e.g. the various GeoDjango extras or the cache backends available or … — I always have to look this up. This is just ticket #28905 which I’m using this opportunity to give a bump to. :partying_face:

1 Like
3

I don’t disagree with the first section Carlton. Although what is perhaps unique in the django-tasks case is the requirement of a third-party package to get the worker to be usable. Your other examples are improvements to Django, but optional in terms of tests and templates, the DB-tasks runner is required.

Now I am not saying we should change this process, but perhaps we give some thought to the documentation so that these upcoming projects become examples for us to point in the future as “the Django way” approach to integrate third-party packages into the documentation. My thinking here is that we start a “how-to” page for future contributors along this route.

In terms of extra-packages, I have written about this before, but currently I think the below:

First I’m keen to see what your proof-of-concept will look like and it also makes me think of draft DEP 15 and how this could allow for extra packages through templated packages etc..

I think it’s also interesting to consider the existence for some third-party packages. Some are created and are obviously never going to get merged into core as they are too niche or add too much functionality etc…

But then there are some who existence is to develop an API that will likely make it’s way to core eventually, django-tasks being an example of this, I wonder if this is an opportunity for us to explore packaging django-tasks into a release of Django without the code living in core? This would widen the queue and allow for quicker release cycles beyond the point of the package being stable, if a developer wanted or a developer just picks up the release packaged with Django at the slower normal release pace.

Either way, there are a few options forward hence the discussion :blush:

4

Yeah, I don’t necessarily disagree with what you’ve said either. (it’s just hard to know in the abstract: the devil is always in the details.)

Yes, part of what I’m working on is about refreshing the How tos around extending Django.

Yes, that’s the other part. What a user perceives as Django when they install it doesn’t have to map one-to-one with what’s in the django/django GitHub repo. Plenty of other projects are developed in a more modular style. The example I give in the talk was FastAPI, for which you install FastAPI[standard]… erm… standardly. (Packaging evolves so maybe they’ll be a “default extra” option in the future.) I’m experimenting with that as a separate package, but as I say, it’s not quite there yet.

All of these things are open for others to play with too. I’ll get there but I’m also working on the various packages I maintain, async stuff in core, contrib.auth, and several other secret side-projects, so I’m not in a rush. (FWIW I don’t think there’s any urgency: Django evolves slowly, that’s a large part of why it’s successful)

1 Like