Django needs a REST story

Django Internals
1

In my DjangoCon US talk, “All the Ways to Use Django”, I had several ideas for how Django could improve as a framework. My first and most actionable idea was for Django to have native support REST APIs. I have been in many discussions about this at DjangoCon US and want to describe the community feedback I have heard on this and give the set of possible solutions and the tradeoffs between them. (Minor note: in my talk, I spoke specifically about JSON APIs, but the discussions I’ve had have been around REST APIs in general rather than JSON APIs in particular, so I’ve done my best to keep this data-format agnostic.)

Why is this urgent? Django is a fantastic web framework but is currently losing in mindshare to other frameworks that focus more on easy creation of APIs (such as FastAPI) and easy support for JavaScript (such as Next.js). While Django can support REST APIs very well via third-party packages such as Django REST Framework (a.k.a. DRF), Django Ninja, there is a perception problem among new developers or developers unfamiliar with Django that Django itself doesn’t solve these problems. This is a challenging problem because it straddles the line of engineering and marketing.

Why REST API’s in particular? REST APIs support a huge variety of use cases, including: inter-service communication, JavaScript frameworks, mobile apps, and external APIs. According to PyPI Stats, Django REST Framework got about 52% the number of downloads as Django itself last month (DRF got 14.4 million monthly downloads vs Django’s 27.9 million). Given DRF has Django as a dependency, this means there are more installs of Django + DRF than of Django alone.

I see there are three high-level ways for us to move forwards on this:

  1. Add direct REST API support to Django
  2. Have Django absorb Django REST Framework
  3. Have the Django docs officially reference community solutions such as Django REST Framework

Here are the tradeoffs I’ve heard on these.

1. Add direct REST API support to Django

This practically is four components:

  1. Serializers/Validation
  2. Class-based API views
  3. Wrappers/decorators for function-based views
  4. Auto-generation of API documentation

Upsides:

  • This would enable Django to create a modern solution that takes the best of DRF and Django Ninja
  • This wouldn’t require Django to absorb all of the baggage of DRF or Django Ninja
  • This may be the only way through this compromise.

Downsides:

  • This requires a plan.
    • Counterargument: we can make a plan.
  • This requires engineering work.
    • Counterargument: DRF and Django Ninja can serve as inspiration for this work, so none of these pieces need to be built from scratch.
  • DRF and Django Ninja already do this.
    • Counterargument: for the sake of new developers, Django should have a native way to do this. Django is inherently modular, and people can still use DRF or Django Ninja or other community solutions even if Django has native support. Django is inherently modular, and just because Django has native support for something doesn’t mean there isn’t room for community alternatives.
  • This could leave people using DRF of Django Ninja in the lurch if those libraries lose support.
    • Counterargument: we can build the new serializers/validators to be pluggable, so that users can continue using DRF or Django Ninja serialzers (potentially with some updates from those libraries). We could also create migration pathways from those libraries to Django’s official REST solution.
  • This adds to the support burden of Django.
    • Counterargument: if this helps bring new users to Django, this improves Django’s longevity as a framework.
  • We could get the design wrong.
    • Counterargument: Django should create mechanisms to experiment a little more. This is what Django contrib was originally intended to be, and there have been ideas to have official Django experiments that are not guaranteed to make it into core.

As to how to implement REST API support in Django, the main thing to decide is how we are going to do serializers and validation. I have heard five main proposals to this subproblem:

  • 1A. Use Pydantic
  • 1B. Extend the existing Django serializers
  • 1C. Make new serializer classes
  • 1D. Use Django Forms
  • 1E. Use Django Models
  • 1F. Handle this as part of the views

1A. Use Pydantic

Django could leverage Pydantic models as the core of its new serialization and validation.

Upsides:

  • Pydantic is popular. Pydantic gets over 10x the number of monthly downloads as Django (421 million monthly downloads), and Django Ninja is an example that people want to use it with Django.
  • Pydantic is supported by IDEs.
  • Pydantic is fast.
  • This creates an easy migration path for people coming from Django Ninja
  • You can make Pydantic models off of Django models (ex. Django Ninja ModelSchema

The big downside I have heard is that this adds an external dependency to Django. This is particularly concerning because Pydantic had substantial breaking changes between versions 1 and 2. However, we can make this an optional dependency of Django, and Pydantic 2 is now used by enough major packages that it is unlikely they will do that again.

This approach would also be better if we added a tool for helping users of DRF migrate their existing serializers to Pydantic-based ones.

1B. Extend the existing Django serializers

Django has existing serializers but they are not as flexible in their parsing as Pydantic and are missing the validation necessary for full API support. We could extend these to make them more flexible and add validation, but this risks breaking areas where these are already used.

1C. Make new serializer classes

So long as we can sufficiently differentiate these from Django’s existing serializers to avoid confusion, creating a new set of serializers could give us all the functionality we need out of serializers without backwards compatibility concerns. Django Rest Framework has a decent stating point for a design if we want to pursue this plan.

1D. Use Django Forms

I originally proposed this as the path forwards in my talk, as Django Forms are very close to what serializers need to do and have built-in validation. There is a recent blog post showing a way to do this. However, Forms have a lot of baggage around rendering that isn’t relevant to serialization that would be confusing to new users. Additionally, bolting serialization on to Forms reduces the future flexibility of both components of the framework.

1E. Use Django Models

Another alternative is to add to Django Models native serialization functionality, such as from_json and to_json methods, and relying on Django Models existing validation. This could also support arbitrary APIs via creation of models with managed = False. As much as this makes it easy for people to get started with generating an API from their models, I don’t like this approach because any complex API will need the layer of abstraction between the models and the API that is provided by a dedicated serializer class. This is especially important for publishing external APIs that need to be carefully curated and versioned. Django in general is set up to help its users future-proof (as opposed do something quick to get started that won’t hold up over time), and I believe its API support should do the same. This approach is also hard to make backwards compatible with DRF and Django Ninja.

1F. Handle this as part of the views

Neapolitan is an example of a library as an example of this. This lends itself well to class-based views, and might be hard to have equivalent for function-based views. This approach is also hard to make backwards compatible with DRF and Django Ninja.

2. Have Django absorb Django REST Framework

Django REST Framework is a fully-fledged, battle-tested solution to this problem that has incredibly wide-spread community adoption. If Django absorbed it into Django itself or included a modified version that was still backwards compatible with DRF, this could be a fairly minimal engineering lift while giving Django native REST support.

A scaled-down version of this proposal would be for DRF to continue as a standalone library but at least get moved to the Django GitHub org like Django Channels.

Upsides:

  • DRF is stable. It has not gotten major changes in many years.
  • DRF has widespread community adoption.
  • This could enable DRF to get more than its current maintenance-mode support. This would also give users of DRF more confidence that it will stay in lockstep with Django and get the same base-level of support as Django.

The downsides to this approach are as follows:

  • We ideally want permission from the DRF maintainers. Such a merger was proposed many years in the past and the DRF maintainers declined.
    • Counterargument: DRF is not under active development anymore so this could look very differently now.
  • DRF is not the only option. Django Ninja is already also an officially endorsed REST library, and Django absorbing DRF would be seen by some as Django “picking a side” in how to do REST API generation.
    • Counterargument: Django is modular. Django having native REST support doesn’t preclude users of Django from doing it another way.
  • DRF is not as performant as Django Ninja. Because Django Ninja uses Pydantic for serialization, which is built in Rust, it runs faster than DRF.
    • Counterargument: DRF is sufficiently performant to have achieved widespread adoption (see the download stats from PyPI).
  • Pydantic models are the future of Python. They are easier to make than serializes and better supported by IDEs.
    • Counterargument: Django has been very hesitant to add third-party dependencies.
  • This adds to the support burden of Django.
    • Counterargument: DRF has substantial financial support, and including it in Django could be a net positive for Django in terms of support costs
  • It’s too late. The time for this was years ago.
    • Counterargument: Django needs a solution to this. Some solution is better than no solution.

3. Have the Django docs officially reference community solutions such as Django REST Framework

I see this as the bare minimum answer. We’re already part of the way there. The Django website now has a community section that includes references to recommended third party packages. These packages have already been approved by the Django Steering Council for endorsement, so we’ve already solved the hardest procedural problem here. There are two things missing. First, the content of this page is not included in Django Docs search; there is a PR up to address this. Additionally, Django should have more guidance to users looking to make a REST API on how to do so and which library to choose so that new users see this less as Django having abdicated responsibility to the community and more that Django is providing its users with multiple options.

Upsides:

  • This is a minimal lift compared to the other solutions.
  • This maintains Django’s neutrality with regards to which style of API we use

Downsides:

  • This is more confusing to new users. This is just a bandage solution rather than native support.
  • This is a departure from Django’s historic philosophies of “opinionated” and “batteries included”.
  • These packages don’t get the same level of support as Django. DRF hasn’t gotten new features recently and Django Ninja is not yet at the level of polish as Django itself. Django is limiting the functionality it can capture and abdicating responsibility for this facet of the framework to the community.

The path forward

First and foremost, we must act. We cannot choose none.

I’m going go post my opinion on what we should do as a reply because I want to build consensus on the problem first and a solution second.

17 Likes
2

I believe we need to move forward with both options 1 and 3. We should add new REST API support directly to Django, and have the Django docs endorse DRF and Django Ninja as the current community standards as an interim solution. Both DRF and Django Ninja have drawbacks, so I don’t see delegating the responsibility of REST support to them as a sufficient answer.

As to how we add REST support to Django, I believe that a minimum, pluggable solution, is the best path forwards. I see either 1A (building new serializers) or 1C (using Pydantic) as the simplest way to achieve this. I am happy to help with the design and implementation of this, but I want to get to some measure of community consensus first.

4 Likes
3

One small comment in regards to point 2. DRF is in the process of being migrated to Django Commons: 🛬 [INBOUND] - django-rest-framework · Issue #188 · django-commons/membership · GitHub

4

I absolutely agree that something needs to be done. I don’t see where “doing nothing” is a desirable - or even a viable - option.

However, I would leave the selection of a solution to others. (I don’t have enough use for a generic REST api to be of much value to me.)

1 Like
5

Just a heads’up this is a topic that has been in active discussions in public and private for the last 5 (10?) years, so a lot of this will have been proposed and discussed, many times. More recently, in the new features repository, search for “REST” or “API”. This is probably the best place to make actual proposals. As noted there’s been movement with the docs, this is pretty intentional, incremental work on a “REST story”.

Also worth a mention – this kind of work doesn’t come for free? The opportunity cost is tremendous. If we choose to do “nothing” or only small steps with regards to REST APIs, it also means our contributors get to coordinate on other topics that are a better fit / more achievable. When we did an (informal) roadmap session with contributors in January 2024, “built-in API framework” was the top #6 item. So there were 5 things that were more appealing to people in that room. And of all the items in that list, REST improvements scored highest on “effort needed”, and didn’t score highest on “perceived impact”. So worth thinking about that too.

Oh last thing re marketing – I’m looking for someone to contribute a “Django ninja intro / how-to” post for the Django blog. If you’re interested please DM me. See Building better APIs: from Django to client libraries with OpenAPI as an example (with REST Framework).

2 Likes
6

I think option 3 makes the most sense and feels like a no-brainer/something we could work on during sprints. I personally wouldn’t want to block that option by attaching either of the other two options to it. :slight_smile:

I also think many (most) people will think of JSON APIs when discussing REST APIs, and it’ll be important to distinguish between the two. Especially so for folks newer to web development. And, by the way, I think we should distinguish audiences by new to web development vs new to Django because those can be two vastly different cases.

But again, I think docs seems like the best place to start there, and returning JSON from a django view is already possible with JsonResponse, so I think making the case for something more elaborate than that in core would important.

It feels like the biggest risk with option 1 would be extra code writing and maintenance for Django, and the end result being many legacy users and projects just keep using DRF, and other options.

Django REST Framework got about 52% the number of downloads as Django itself

Could be just as much an argument that almost half of the people using and trying Django don’t need a REST API. But it could be interesting to update the docs with ‘official’ recommendations and see if that number budges at all.

Even more of a tangent, but I wonder if comparing to other frameworks with similar level of resources would be instructional? I’m out of the loop but I wonder if similar frameworks in PHP, Elixir, others that are also battery included also include REST apis as part of the batteries?

Those are my thoughts at the moment, hopefully they make sense :smiley:

2 Likes
7

I’d argue that this would be worth putting in a budget and some financing around to get it done, well and quickly.

8

I want to note that these three options are not mutually exclusive. All three could happen.

I think option 3 (better Django docs on how to do REST via community solutions) is a sensible starting point, as it’s not a huge lift. It would be awesome if that can get some progress during DjangoCon sprints.

9

DRF is already documented as part of the “Ecosystem” here: Django Community | Django

As for an official recommendation? DRF has its problems and I would just caution as to recommending people use it because it definitely comes with dragons. Don’t get me wrong it has some nice pros as well… however once a project gets stuck with something like DRF it’s nigh impossible to switch it out. I’m personally seeing a lot of my trusted colleagues opt for something else like Django Ninja for eg.

Also not to say that Tom & more recent maintainers Carlton & the rest of the community haven’t done a great job & kudos for their hard work but I’m sure in retrospect there would be some changes made to address some of the more common criticisms (performance, design choices).

10

DRF’s view layer bit is fine. It’s still great. (There’s an adrf project that gives async versions of the views there, though I think that’s a distraction TBH.)

The issue with DRF is that the serializers are last generation. They’re slow compared to modern alternatives. Pydantic is the poster child there but there are numerous options in this space which are competitive (better even, IMO). Cattrs and msgspec are the two that are (maybe) the most interesting. For disclosure I use cattrs in all my own work, and am a very happy customer.

I’d be reluctant to endorse a particular serialisation option as we stand today. It’s an active area of experimentation and development. There’s no one clear winner that we should crown.

(This is similar to template component libraries, and frontend integrations: Folks want to be told what to use, but it’s too fertile a time to say. Historically, it’s also proven to Django’s strength not to have made a commitment to some frontend framework that then fades, as they do.)

That doesn’t mean we can’t improve the ecosystem page (and give it sub pages if needed) to let people know what their options are now. Indeed I think we should. I don’t think it’s unreasonable to maintain a small number of topic guides that aren’t timeless — i.e. that would need maintenance over time: that was the whole point of making the ecosystem page a steering council responsibility.

That the ecosystem page still needs to be integrated with the search is, yes, a good point, but it’s already in progress. Pushing that, and then evolving what we started seems the way to go. (That’s sort of Option 3. We’re already doing that, but work remains to be done.)

§

In terms of Option 1, I think 1c is the route forward. Modern serialisers are what we need. I don’t rule out such being based on Pydantic — although that wouldn’t be my choice — but I think there needs to be a more Django-sympathetic layer in-front of that. [More to say here, but not today…]

I’m not sure what’s meant by 1f — it’s not a view-layer question to my eye. Views will use serialisers whilst turning requests to responses.

And here’s my usual tune to close… :musical_notes: Any option should be prototyped as an ecosystem package. Get the quick proof-of-concept down, put it out there, and let’s play. There’s no reason why we need to block here waiting for a committee decision that will never come pending a proven solution that we’re yet to build. :drum:

Thanks for raising this again @zags.

2 Likes
11

I think the docs solution is less about a single library recommendation and more about that you can do REST API’s in Django.

Rails does a good job here IMO: Using Rails for API-only Applications — Ruby on Rails Guides or Laravel: Eloquent: API Resources - Laravel 12.x - The PHP Framework For Web Artisans the focus here is not which package, but that Django can do REST APIs

If I had to pick a preference in terms of code I would love to see something that would one day work for both API’s and form’s (ie the validation of DRF and Forms is very similar). So I would say 1c with a long term view to allow for Django forms to be integrated into the new solution.

2 Likes
12

Yup 100% … that’s my main issue. It really compounds too. I’ve worked on some projects where it’s just feels a “serialisers all the way down” type of deal.

Eg I recently setup Simon Charettes handy django-seal. Trying to get the serialisers-all-the-way down to comply with my no n+1 demands was tricky.

13

Yes, that’s the trick.

This is kind of what I mean by a Django-sympathetic layer — the ideal here is something that’s query aware to avoid lazy related field fetches with nested serializers (whatever they may be).

1 Like
14

Been thinking about this for a few days.

First off, I agree with Carlton that serializers feel previous generation. I also like the idea of, what Carlton calls a Django-sympathetic layer.

Ideal World: For me, that would mean that I could choose to use Pydantic under the hood but be able to swap it out in the future for other options. What some of the frontend framework world has done (GitHub - standard-schema/standard-schema: A standard interface for TypeScript schema validation libraries) is interesting.

This would be more work though, because you’d essentially have to take arguments to some serializer fields and (maybe lazy runtime serializer cache) map those to say, Pydantic under the hood as an adapter layer. This impacts performance as well.

Practical World: If Django Ninja was open to it, bringing that into Django (or a similar variant) as the official solution, and just being okay depending on Pydantic. A lot of the Python world is now used to Pydantic in one way or another, and if Pydantic really has all these organizations (Why use Pydantic - Pydantic) using it, I doubt they can make a huge breaking change leap from say 2 to 3 as they did from 1 to 2.

15

I would be excited to see, support, and work on option 1 in some fashion. I think that is a long term solution, and in the meantime supporting what is available through more content, number 3, seems like a good approach, although maybe with mixed signals.

This is a great call out, and thanks for sharing the list. Looking at the top 5, it seems like progress has been made on a number of those. So maybe that means it is time to start looking at number 6 :wink: ?

I wonder if there is a way to break the problem down a bit more into iterative steps? For example, thinking about this from Carlton’s perspective of “validation with a Django-sympathetic layer,” I can see at least three pieces that may be needed based on my experiences.

  1. A way to mark views as API resources. Like the routers in both the DRF and Ninja paradigms. This is what builds the basis for support of introspective features like documentation generation.
  2. A serialization/deserialization framework.
  3. This Django-sympathetic way to get the serialized data

The router seems like a concept that could be written into Django.

It could be interesting to explore if the serialization/deserialization framework can work based on pluggable backends: one for Pydantic and another for cattrs, for example. I think this might be what Micah is getting at as well. This could be a layer that isn’t yet query aware but allows for data sanitization and error handling, at the least, with the idea that this data would be passed onto the final Django/query aware layer from there.

It is possible we could build those two pieces without fully fleshing out the Django sympathetic layer of it to start, and this would get us some level of progress with functioning views, even if they don’t have the extra Django touch to them yet.

I’m just throwing ideas out there on this, and I’m curious if others, with more experience than me, think this could be a viable approach.

My last thought is to caution against absorbing Ninja as it is now. I use it in production, and I do like it, but it has some rough edges. The project is also struggling with maintenance and already behind on issues and PRs. There has been a fork, django-shinobi, with an aim to be more active.

3 Likes
16

Yeah. This.

I’m going to put my cards on the table. As I intimated above…

… I think Django-Ninja is great. I’m glad it exists, and I think it showed that a different approach is totally available. But I don’t think it’s the final thing, and I don’t think we should just pull it in.

In the short-term, advocate alongside DRF as an available option, as per Option 3 in this thread, by all means.

1 Like
17

hi
i’m sure this was considered, but just have said it

if there would be a django layer in fron of the serializer, then maybe the serializer can be made swapable, and have django interact with whatever serializer developers provide
and django can have a default option of it’s own (or no option, and put the borden on 3rd paties)

2 Likes
18

While I don’t have a lot of technical suggestions to bring to the table, I do agree that adding some form of opinion on community options regarding REST would be of great help.

From a purely anecdotal perspective: when I was first starting, there was a time when I definitely wondered about how people were finding ‘the good projects’ with regards to what integrated nicely and what was ‘best practice’ kind of a thing. Having a place where the community can go - even if only to see suggested third party options - I think would go a long way towards making things a little easier on newcomers (as well as well seasoned devs). New folks could see ‘what is good’, older devs could periodically check to see is there something better out now.

I would love to see some integration of it within Django, but only for the sake of making something already amazing, even better for the community at large.

2 Likes
19

I appreciate the energy and thought you put into this. Django is an amazing framework, one of the best! I agree that not having a better REST story is starting to be a problem for the reasons you mentioned. I’d support #3 in the short term so new developers immediately see Django’s API story, and plan towards #1 or #2 as a long-term goal.

#2 has a lot of merit since a collaborative effort would benefit the whole Django/Python ecosystem. If DRF continues to lose maintainers, the financial support to DRF could shift to other places, really stressing the need for a better REST story in Django.

On the technical side, I have some hesitation about Pydantic:

  1. From my experience with FastAPI, it felt like I had to learn both FastAPI and Pydantic when I made my first FastAPI project. This can be a big lift for newer developers who are already wrapping their heads around Django. If we include it, it should be done in a way that feels “batteries included.”
1 Like
20

I just read this at lunch and was compelled to respond. I’ve been writing Python, using Django, since 2008.

Time and time again I’ve seen micro frameworks (Flask, Falcon, etc.) peel off segments of the developer market share, and the biggest reasons I’ve seen this happen have always been the perception that 1) their project will perform better (better speed and throughput), and 2) the micro framework will be “small/clean/simple” and will "stay out of their way”. These decisions are often made by amateur and mid-level developers (starting a new side project, want to try a new framework, etc.), and sometimes by misinformed senior level developers who think that they have been missing out on something. There is a third cohort, those who actually greatly benefit from a micro framework, such as somebody doing something like game development or other real time things, especially if they can’t make good use of Django’s ORM. There is always a best tool for a given task, and sometimes Django is not that. However, most of the time, if python is the language and the web is the domain, Django is the best tool.

With all that said, the 1 reason above (performance) will not be helped (benefiting Django) by adding another tool trailer to the back of this caravan. The 2 reason (minimal, stays out of my way, clean, etc.) will be negatively impacted (harming Django) by adding a tool trailer to the back of this caravan.

IOW, being more opinionated about highly divisive topics, such as the correct approach for REST, etc., is not going to make people want to switch to Django. Batteries included does not mean "we do all the things”. Batteries included means “we do all the things that you’re most likely going to have to do anyway and where you are unlikely to have much of an opinion about how it’s done”. Even to the degree that the latter has been achieved, a lot of backpedaling has been done over the years to make those items more extensible, because even the most “obvious” things, such as session management and forms, have multiple solutions that each developer / team has reasons for preferring over another.

I, for one, despise DRF, and I have despised it since the first time I had the misfortune of using it. I can’t stand that it re-implements much of what Django already has, instead of relying heavily on the existing model/form functionality. It’s like a spaghetti mess of spiderwebs. I prefer Django Ninja by a longshot, even though it also has many warts that drive me crazy and make me think "I would jump ship immediately if another REST tool came out that worked similarly but solved all these warts”. That’s the beauty of being able to plug in things that you want. The batteries are not the end product, the batteries are the minimum necessary pieces, and REST is not a minimum necessary piece to start developing a web application, even if DRF happens to be extremely popular still.

I would even go a step further, suggesting that DRF becoming the standard in the Django world has been one of the prime reasons that people have switched to FastAPI. I personally learned how to use FastAPI, before I learned of Django Ninja, for this reason, and I have worked with multiple people who switched to FastAPI because they can’t stand DRF. Making DRF part of Django or even recommending it as “the solution” for REST would be a grave mistake.

It is a grievous non sequitur to lose a bunch of market share to FastAPI, and glean from it that the vendoring of DRF, the furthest thing from FastAPI that we have, and which has been out of active development for years and which does not use Pydantic at all (one of the other primary reasons I’ve seen people switch, strong typing has become a preference for many), is the answer.

If we want Django to have better REST and other API support, finding ways to simplify the core components that are required to piece these things together is the answer, not tacking another rusting trailer of old tools onto the back of the caravan.

In practice, this means spending time figuring out how we can make it more performant and easier (for developers) to implement REST solutions (like Django Ninja, etc.), such as adding the necessary low-level API changes to things like models and/or forms, making it easier to translate ORM models into Pydantic models, etc. Secondarily, if there were higher level requirements, putting those into a contrib app (perhaps which requires Pydantic, or not), so that it could more easily be removed later or made an external package if necessary, would be a reasonable next step.

Put more simply, I am a strong -1 on this, for what it’s worth, from the perspective of a multi-decade Python engineer that has been using Django since circa 0.96.

2 Likes