A Path to Modernizing APIs in Django (DEP 0017 Content Type Parsing)

Django Internals
1

In the most recent Django Chat episode with Haki Benita the topic of APIs in Django came up. @carltongibson mentioned, “There’s a long standing DEP that’s been stalled for years about introducing content negotiation into the request object and modernizing the request object.” That is DEP 0017, which is in progress.

@carltongibson 's basic point is we need to update the underlying primitives before doing anything around a recommended API in Django core.

What are the blockers?

  1. Modernize HttpRequest: see DEP 0017.

  2. Serialization: Django needs a modern, performant, preferably pluggable serialization system. This is the biggest technical challenge.

  3. API view layer: Relatively straightforward. Writing views and decorators isn’t the difficult part.

@hakib’s perspective was: establish the official API interface first, then optimize its performance. He sees the lack of a standard API experience as more pressing than choosing the perfect serialization implementation.

I lay all this out here because I think DEP 0017 needs all the support it can get–thank you to @carltongibson and others who have weighed in–and I know we’d all love to see Django update its API story.

The full podcast episode talks about django-ninja and other more-modern implementations so I’d encourage everyone to give it a listen. The API part starts around 22 minutes in.

some people know

6 Likes
2

Thanks for posting @wsvincent.

FWIW I think progressing DEP 17 is the hard bit. Getting to yes, we will actually modernise the request object, and add the content-type aware body parsing is most of what’s missing.

Serialisation has various great options, be it pydantic, attrs/cattrs, msgspec, or something else. There are lots of modern options, and they’re all in the ballpark of very fast. All of them enable OpenAPI schema generation quite trivially. My django-mantle project, that uses attrs/cattrs, shows how this can be ORM-aware, which I think is worth aiming for.

The view layer is, yes I think, pretty simple. Allowing flexibility about your choice of serialisers would be important.

And then there’s niceties, like permissions, and rate-limiting, which I think any in-core solution should lean on the ecosystem, and again be flexible, in Django fashion.

But I don’t see these as killers.

tl;dr: If we had a modern request object, we could probably have a new option for APIs weekly — We could have a Do an API tutorial, even. (As Adrian Holovaty said when we had him on the podcast There’s no THERE there at that point. )

4 Likes
3

Thanks for starting this discussion, Will!

Speaking of the need for a modern, pluggable serialization system, a few months ago I proposed making Django’s internal JSON backend configurable, keeping the current one as the default while allowing projects to use alternatives.

My proposal focuses mainly on JSON encoding and decoding, rather than the object serialization for APIs you’re discussing here.

Still, I wonder if this work on modernizing Django’s APIs could also be a good opportunity to consider a pluggable JSON backend in Django core, as a separate but potentially complementary improvement.

I’d love to hear what you both think!

4 Likes
4

It’s certainly related, and I think that’s a good proposal that we should advance.

2 Likes
5

hi :waving_hand:
I’ve opened these two issues (tho probably could use some rephrasing and explanation)

and

ultimately, i think django’s auth system should become more extendible and give auth backends more power, so auth methods like JWT and tokens would work, and things like csrf can be handled based on need

6

i also think we might find an abstraction for openapi to be useful
there are a lot of openapi libraries in django ecosystem, but none of them work with django views

and there are a lot of serialization libraries, each has it’s own way to generate openapi for

so a pluggable interface that accepts different backends could be useful, similar to how caching and tasks are handled

2 Likes
7

Yep, this is a good idea.

There’s two kind of general steps:

  1. The request/response body shapes, which your serialiser library of choice will have its way of giving you.
  2. The endpoints and operation parameters, and all the rest of it, which is Django view specific, and needs walking the URL conf essentially.

A shared core for 2, with “plug your 1 in here” would be amazing.

2 Likes
8

How much would it cost to get DEP 17 done?

9

It needs someone to pick it, go over the history, and resolve the outstanding issues, or at least have a plan for that.

I’ll just add this, because I keep getting asked:

Personally, I ran out of energy to keep pushing here. It’s on my list to continue this work in a third party package, to demonstrate the solutions, and to be able to iterate quickly.

But that’s pending me needing to write an API. I started the new work project four years ago now, and we went HTML first and still didn’t get to adding an API. I thought I was going to add one this summer, but we ended up just adding a single MCP endpoint, which uses Mantle for the serialisation layer, but didn’t take on the rest of it at all. So, it’s still pending. Eventually I’ll get there, and if no-one has done anything by then, yes I’ll pick it up.

But I’d much prefer that others (who have a more pressing need) pick this up. My time is elsewhere at the moment.

2 Likes