I’ll be working on the project “Implementing a Formal Experimental API Framework for Django” along with @nanorepublica.
This project aims to create a safe space to experiment with features. Users can opt-in to test these and community can iterate on API designs based on actual feedback.
For DEP 2 rewrite and implementation, I am looking into how we can safely isolate experimental features.
So, beyond namespacing and settings-based flags, are there other methods we should consider?
Is an installation barrier (using package extras) like pip install django[experimental] a necessary safety measure to protect stable releases, or does it introduce too much friction for developers who want to test new features?
I don’t think we’ll arrive at a standard set of rules for every possible situation. I suspect each feature will have its own reasons for being added in an experimental state, and therefore it would have its own reasons for when it would exit experimental. I think it’s fair for the community to ask for a feature to include why something qualifies as experimental, what are the criteria that needs to be met for it to be made stable and what are the criteria that needs to be met for it to be removed entirely.
One thing that could help those discussions along are guidelines for what are reasonable justifications for experiments and what are some example/plausible exit criteria.
Okay, that makes sense, letting the feature proposer justify the reasons for why the feature needs to be experimental along with the promotion/purge criteria will make the process flexible for different situations.
Since a big part of this project is integrating the experimental framework into Django’s existing contribution process, I was looking at the django/new-features repo for inspiration. They scale their requirements based on feature size (Days to Weeks, Months and Quarters). We could use that same sizing logic to provide the guidelines and examples.
Ultimately, these guidelines will depend a bit on how the experimental API framework is implemented. So, do you have any ideas on how that should be implemented?
It feels like if we’re talking about pip install django[expiremental] then we’re just creating a third party package which is no different than what we do today. I think that’s fine, but lacks ambition for what we should try to support.
I suspect, we’re trying to account for changes we can’t do in a third party package, but don’t want to commit to backwards compatibility. For that I think it’d be a mixture of having a dedicated package in Django like django.experimental or experimental modules in various modules, like django.contrib.auth.experimental. I prefer the package route rather than modules because I don’t expect this to be used much and it would allow us to see all the experimental stuff in a single area within the code.
I’m guessing at some point we would need a setting too. For example, if we wanted to change something about url resolving. So maybe we should have settings with the EXPERIMENTAL_ prefix?
I think there’s some appetite for an actual feature flags solution to be part of Django itself. This isn’t a minor addition so it’ll probably need the DEP process, but as an actual code area, it might be something you want to spend time on.
Whether we use such on bits of core itself is a separate topic. The Why here needs fleshing out. We already have a process: put it in a third party package, and then it possibly progresses to core. This system works: just as recent examples, template-partials, tasks, and fetch modes all went this route. Subatomic is doing similar.
Why do we need more? Not saying we don’t but I think the case for why is really not laid out.
Is the problem here social? Is it about messaging and communication? If so, we want to be careful that we’re not trying to implement the wrong kind (i.e. technical?) of solution. Is making the new-ish Ecosystem page on the site much better in fact the project here?
Stability is Django’s main selling point. (Reportedly, David Lord (Flask maintainer) was praising it just at PyCon US recently.) The danger is we break that. So how do we assure we don’t?
Of the routes suggested in the OP, I think the extras idea is the more feasible: we need experimental features (whatever that means) to not be tied to Django’s release process. (So is this just a way of blessing some packages that are in the pipeline? Again, is that really a comms issue?)
But extras or flags look like two different projects, not one.
(Just some initial thoughts. Excited to see where this goes!)
I largely agree with Carlton here. The one area where I wonder whether the third-party approach is suitable is beginner-focused features - for these we want the barrier to entry to be as small as possible. But maybe documentation can also address that?
I agree that third party packages work great for features that sit on top of Django like template partials or tasks. However, this project is meant for deep core (like work around url resolver) that are technically difficult to implement as a third party package.
Node.js(experimental flags) and Rust (feature gates) use this built-in approach specifically for these types of deep features. Improving the Ecosystem page is a great social goal for third-party discovery, but it doesn’t solve the technical need for a safe way to iterate on these internal core changes.
Sorry, I should’ve been clearer in my OP. Look at them as different strategies for isolation.
Logical Isolation (Direct Core): Experimental packages (like django.experimental or modules like django.contrib.auth.experimental as suggested by tim) and EXPERIMENTAL_ settings are included in the standard Django install. This maximizes discovery and testing but relies on the developer to respect the ‘experimental’ labels.
Physical Isolation (Package Extras): Everything in Path 1 is placed behind an installation barrier. This changes the pip command, which provides an extra layer of safety and explicitly signals to the user that they are entering an ‘Experimental Zone’ where stability isn’t guaranteed.
The goal is to decide if the extra safety of package extras should be added or not. Using Extras for isolation is a proven pattern in the Python ecosystem for maintaining lean, stable cores. Pydantic uses it to manage officially supported but optional features (pydantic[email]), and TensorFlow uses it to isolate complex, environment-specific logic (tensorflow[and-cuda]).
I want to build a framework that can support different implementations (namespaces, modules, flags or something else) we eventually decide on, but first we need to agree/disagree on the use of package extras.
To ensure we don’t break that, the process would involve developing and testing these ideas as a third-party package first. By using these isolation techniques, we ensure that a Django user who doesn’t opt in for the experimental framework, never executes a single line of it.
That’s a fair point. While the extra install step is a small barrier, I believe the trade-off is worth it to ensure beginners don’t accidentally break their environments with experimental code. Great documentation is definitely a major focus of my project to bridge that gap. The first two points in the “Success Definition” Andrew @nanorepublica provided for this project specifically focus on updating and maintaining high-quality documentation, so rest assured he won’t let that slide!
I think we need to go a little more slowly to answer the Why question here.
That’s exactly what needs to be shown.
This isn’t a good example. To swap in a custom resolver you’d just override resolve_request in a custom handler, and use that in your application. It’s perfectly doable in a package.
Almost without exaggeration, every one says, “Oh my feature needs to developed in core”. Once they realise that’s not happening, they then do the third party package. (I can immediately think of several cases of that happening just within recent cycles.)
It strikes me that the actual number of cases where a package isn’t actually viable are so rare as to not merit a whole “experimental features” framework. So, I come back to thinking, that’s not really why we might want one, or where the benefit really is.
That why is still sat there.
There’s potentially a major change to the way Django does things being suggested here. I think it’s important that get the rationale right. (Otherwise we’ll have little chance with the details.)
When thinking about experimental, I’m thinking about landing code iteratively in Django.
I have not shipped or reviewed code in Django so my comments are based on my experience in other projects. The current process of landing things in one big pull request might be creating a toll on code submitters and code reviewers because each review cycle contains the whole shebang. By using an experimental namespace, some larger features could land in smaller batches that are more manageable for humans to write, review, update.
Only when all the pieces have landed can they be moved to a non experimental namespace. This part is creating extra work but it might be worth it if the rest of the work gets easier.
Such a process could also reduce stress when shipping major versions. A new feature with issues requires work to either fix them all in time for the release or to be backed out (which can be difficult to do cleanly). With an experimental namespace, none of that is needed.
This is the principle of Continuous Integration:
Integrate code changes frequently to ensure that the integrated codebase is in a workable state.
The longer development continues on a branch without merging, the greater the risk of integration conflicts.
People that participated in landing CSP and Tasks might have thoughts about this idea.
UPDATE: This process is independent of that code having started life as a third-party package.
I agree with Carlton here. Django is fairly good at inheritance and fairly stable even in its private APIs.
Off the top of my head I can’t think of any feature that couldn’t be built as a 3rd party.
I also fear that two different deprecation pipelines will confuse users.
As a maintainer, I am eagerly awaiting annual release cycles and a consistent clockwork in which to adapt to changes. This would actively work against that.
The only thing in the past that didn’t work without changes to Django core was ASGI. But we managed, and all subsequent async transformations are so stable that I haven’t noticed any problems with the deprecation cycle.
That being said, I wished for some packages to have broader adoption before merging them into core. Mainly the task framework, which isn’t bad, but also wasn’t very mature compared to its alternatives like Celery and even Dramatiq. It’s a bit painful to reach this maturity in Django, but this isn’t an issue with the deprecation process either.
I am wondering if experimental flags ought to initially be limited to experimenting with changes in the existing core codebase. New features can rightly be trialed in a new package.
A well known example here is the auth.User model and it’s first/last name fields. The current general expectation for a User model is either to use the default one of have a custom one in your project. I know of 0 common third-party apps providing a custom user model to a project. While I could see a prototype to experiment with a new User model in a third-party package I wouldn’t expect much uptake due to the mentioned expectation.
However I could see the use of an experimental namespace to play around with what a new User model could look like inside Django, free of the need to maintain backwards compatibility for a period of time. This would give folks the opportunity to experiment with a new breaking API, from which we get to a known end state, then working out the migration steps becomes potentially easier? Yes this could be a third-party package, but when the API has already been established to be inside Django ought it not stay there in some manner?
Personally I’m doing this with contrib.messagesas I think a refactor would improve the internals, so I am playing around with what the end design of the API looks like, then I can propose steps to get there. Again having an experimental namespace (or similar) would allow folks to opt in to trying this out.
This isn’t a good example. To swap in a custom resolver you’d just override resolve_request in a custom handler, and use that in your application. It’s perfectly doable in a package.
I think “It’s perfectly doable in a package” is true as long as the expectation is that the 3rd party package may need to do some monkey patching. For example, if you dig into what changing resolve_request might look like if you were trying to add method routing, one thing you might see is the resolve method would need to change on URLResolver. URLResolver is called internally by get_resolver which doesn’t provide a way for the user to customize URLResolver. Perhaps this could be solved through “public interfaces” but I couldn’t see it.
I say this playing devil’s advocate though because I actually agree with your point. If I were to go back now and try and finish this pr for myself, I’d do it as a separate package first just monkey patching whatever is needed in django so I could just use it.
If we’re searching for rationale, maybe monkey patching = bad is something?
You’d just override BaseHander.resolve_request (src) and use your subclass in your wsgi.py/asgi.py file. It’s trivial. (The hard bit is the implementing it )
monkey patching = bad
I take it that subclassing isn’t monkey patching, so this thought doesn’t apply here. But we’re talking about implementing experiments. Even if you hold that monkey patching is bad, presumably for stable code, being able to swap out an implementation to show what’s possible is going to be a standard approach whether we have a flag solution available or not…
if flag: apply_monkey_patch()
One of the great things about Python is that code is just one more type of data.
I take it that subclassing isn’t monkey patching, so this thought doesn’t apply here.
Absolutely agree: subclassing is not monkey patching, and django does provide a hook here that can help with some custom implementations here.
You’d just override BaseHander.resolve_request (src) and use your subclass in your wsgi.py/asgi.py file. It’s trivial. (The hard bit is the implementing it )
You can ignore me if you understood my original point. But if not, I am going to try and explain it again because you used the word “just”. A challenge is not about finding hooks like subclassing but about dealing with code that might not have a hook available for the user to override without patching. Yes, like you having pointed to, there is a method we can override to implement our own implementation of resolve_request. Other places in django call get_resolver. When my new HttpMethodHandler calls my new HttpMethodURLResolver, but the rest of django calls the framework’s URLResolver, something is not going to work as expected. Patching the URLResolver in django can be a start to implementing this feature, but I think there are other places in django that would need to be patched in order to implement http method routing as it is in my pr today (the real proof would be in a working third party package that I didn’t make, but I am trying offer my best perspective).
@massover Yes, you’re right — to do it fully you need to patch get_resolver. My assumption is that any resolver would need to be compatible with the existing one, and that it’s during request resolution that performance gains will be assessed.
But it’s somewhat moot. You think it’s a better example, I think it’s a worse one. OK, we can’t progress there.
The point is more about the claim (that’s always made) that we simply must develop something in core. It’s basically always false. You might say monkey patching is bad, but it’s just Python, and for experimental features it’s fine. We did some horrible things patching template loaders for template-partials, but it was fine. Folks could assess the proposal on an opt-in basis. We could fix the issues, unbound by Django’s release cycle, and then when it was proven, merge it.
I repeatedly hear desire for something in this space — to make the experimental features story better — but I’m sceptical that it’s a technical problem. (Third party packages are fine.) So I am concerned that we’ve not yet really identified what the goal should be, and that why I mentioned at the top.
I actually intended to use my example to agree with you given the original discussion was around url resolver. I wanted to offer some insight as to why I (someone who doesn’t contribute to django) might have opted to make a pr and a dep (lol in retrospect), instead of just realizing “oh i can just patch django in a third party package to do what i need now”, and that’s the more realistic approach to getting it merged if that does make sense later anyway. I’ll stop over explaining now though, sorry for the distraction!
I see this not as a change, but as an addition to existing methods, all current methods and procedures remain fully intact. The goal is to provide a structured, native pathway for features that would benefit from real-world usage and feedback during their design phase.
This ticket has been stalled for 9 years. The PostgreSQL and MySQL implementations were ready years ago. However, because Django currently requires simultaneous perfection across every supported database, the entire feature was on hold because of complex driver failures on Oracle.
A third-party package is not an option here, as Mariusz Felisiak noted in the PR review, proper ORM integration requires native as_{vendor} hooks inside core, rather than fragmented if/else vendor checks.
If we had an experimental namespace, this could have been landed in the smaller, manageable batches. The stable Postgres and MySQL pieces could have shipped years ago behind an experimental flag and the Oracle implementation could have been iteratively fixed over subsequent releases.
An experimental namespace provides a native pathway for features to mature through more usage and feedback. We see the need for this in Ticket-13240, where a patch was marked Ready for checkin but stalled because maintainers couldn’t agree on the API design. Alex Gaynor felt the proposed design would be confusing, but without a way to gather real-world feedback from users safely, the design debate reached a stalemate and was abandoned.
@nanorepublica raised a great point about established APIs, when a feature is already central to Django core, a third-party package isn’t a viable solution. We see this exact problem in Ticket-10227 regarding reverse OneToOne relationships. This has been open for 17 years because changing the reverse lookup behavior to return None instead of throwing a “Does Not Exist” exception would break backwards compatibility for legacy apps. An experimental namespace is the only way we can experiment with a new breaking API for established core systems natively, letting the design mature safely in place.
We see an even deeper structural need for this in Ticket-24312 (making it possible to import models safely at any time). Rewriting Django’s app-loading system to remove heavy reliance on import side effects is a massive architectural shift. A third-party package cannot solve this because it is bound by the very loading system it tries to fix. An experimental namespace provides exactly the playground @nanorepublica described to map out what the end design of such a breaking, core refactor looks like without breaking stable production environments.
it’s fair to ask if the problem here is social, better communication and ecosystem are definitely needed, but the “why” of this project is to solve cases where social consensus is impossible to reach without real word usage and feedback, if the problem was social then these tickets wouldn’t have been around this long!!
All in all, the more I research this space, the more I see how many use cases it can serve from safely navigating breaking API changes to building new features alongside the community. By gathering feedback and usage data from real-world projects, we invite more suggestions which eventually leads to more active contributors. This can possibly turn innovators and early adopters into active contributors and motivate others to switch to latest versions to experiment with new features.
)