Django needs a REST story

Django Internals
41

Regarding documenting what is available, there was a general call for Django Ninja content during DjangoCon US. Based on that request, I put together draft here based on some of my experiences and learnings using Django Ninja in production this year. I’m interested if something like this might fit into a guest blog post on Django or fit well into ecosystem documentation. I’m open to feedback on it.

I’ve been tinkering with some ideas around routing and protocols since this post came out, and I’m happy to share some of the early ideas here specifically on the protocols-draft branch. I’ve mostly been working on sketching out the interfaces here, while still ensuring that the concepts at least kind of work up to this point. I’ve tried to come up with a way, if not the final way, of incorporating routing, parsing/rendering, and mapping to Python objects into a cohesive API-first approach without being too opinionated just yet. I used Pydantic and DRF Serializers as my examples because I’m most familiar with those, but I think things like cattrs and msgspec could fit in as well.

I think there is still work to incorporate a Django sympathetic layer into this proposal, and that needs some more thought. I agree that is the more complex part of this initiative with the most possibilities. This approach is very function-based so far because that is what I am preferring these days. I think it can hook into CBVs as well, and that is something I’m hoping to explore more.

This is my first go at it, and I’m excited to see what other approaches others might bring to the table.

@lisad in my day-to-day I tend to agree with a lot of the concepts you are proposing around “stable APIs.” I think of it more as “explicit” than “stable,” which you could argue is not really different. For example, in today’s ecosystem, you can still using ModelSerializer (DRF) or ModelSchema (Ninja) with an explicit fields that isn’t set to __all__ to keep your APIs explicit without necessarily redefining each fields type from your model. With large payloads and performance critical APIs, I often just use functions for building TypedDict instances since then my IDE can still help me, which is similar to your functions concept. The idea of using templates has come across my mind as well, but I prefer having my API mapping defined in the same file as my API function, so having a separate template file is not very appealing to me, personally.

I have also really grown to like the explicit nature of decorators on my APIs, but we’ll have to see if that is a fad for me :grin: . My example implementation definitely leans towards the decorator concepts. The API error handling on each of the views felt like it got a bit noisy in my experimenting so far, but I’m still playing around with that one specifically.

4 Likes
42

I enjoyed reading your draft @camuthig. I think @thibaudcolas is leading the effort for the blogs, but it seems on topic to me! :wrapped_gift:

43

@camuthig I fully agree.

When I say “think in terms of templates” it’s maybe more of a vibe than a specification.

In my current main project, we have views organized by access control (public, users, staff and api) so they’re in 4 different files, but the template-y methods for the API views are in there with the view methods.


@api_get
def entry_info(request, entry_id):
    entry = RegistryEntry.objects.get(
        Exists(Service.public_objects.filter(pk=OuterRef('service_id'))),
        pk=entry_id
    )
    verified_domain = entry.get_verified_domain(entry)

    if not verified_domain:
        raise ValidationError(f"No verified domain found for {entry.service} ({entry.service.company})")
    return entry_template(entry, verified_domain, request)

def entry_template(entry, verified_domain, request):

    return {
        "trustLevel": entry.trust_level,
        "trustStatus": entry.trust_status,
        "verifiedDomain": verified_domain,
        "trustInfo": _trust_info_partial(entry),
        "serviceInfo": _service_info_partial(entry.service, request),
        "operatorInfo": _company_partial(entry.service.company, request),
        "authConnection": _auth_connection_partial(entry.service, request),
        "dataConnection": _api_connection_partial(entry.service),
        "entityDataValidity": {
            "validFromDT": date_field_to_utc_string(entry.approval_date),
            "validUntilDT": date_field_to_utc_string(entry.expiration_date),
        },
    }


def _trust_info_partial(entry):
    partial = {
        **value_or_not('servicePrivacyPolicy', entry.service.privacy_policy_url),
        **value_or_not('operatorPrivacyPolicy', entry.service.company.privacy_policy_url),
        'termsOfService': [ url for url in [
            entry.service.privacy_policy_url,
            entry.service.company.privacy_policy_url,
            entry.service.company.terms_of_service_url
        ] if url is not None],
        'dataProtectionOfficer': _dpo_partial(entry.service.company),
        **(entry.trust_info or {})
    }
    return partial

...

I think most of this is as easily readable as a serializer that has an allow list, and it allows the structure of the API to be designed specifically for JSON. Django models often flatten data when it’s going into SQL, because you might need to filter on a field so you can’t stuff that field deep into structured data. but we don’t necessarily want the same flattening for data being organized into our APIs.

This more extended example shows some of the places that have rough edges. Eg The idea that the ‘servicePrivacyPolicy’ field can be excluded if the entry.service.privacy_policy_url db value is NULL is something that could be better presented.

1 Like
44

Using this topic as an aggregation point for ideas, I want to plug @emma ‘s djrest2 project as another conceptual lead. She also wrote a post thoughtfully breaking down what is needed for an API versus a HTML based request-response flow. In djrest2 she uses the generic View CBV as a basis for building lightweight JSON-based CBVs and uses Form for serialization and deserialization.

3 Likes
45

I penned some thoughts on this today, which expands a bit on my previous post in this thread Django & REST & APIs - Software Crafts

1 Like
46

I’ve been considering @nanorepublica ‘s thoughts regarding treating this as an exercise in building an API (as in Python interface, not the REST kind).

Trying to think about the REST story as an interface/protocol led me to think about the serializer/schema/form layer as a scoped concept. I know nanorepublica called out not just focusing there, but their example is what made me think of it. Specifically I was considering what the implementation of the serialize_one function might look like and how it might be extended to also work well in function based views. This led me to consider the role of the Form class in HTML development and how it might help guide design decisions in API development. As shown in djrest2, Form can provide a solid basis for JSON, however, there is a lot in the class that is specific to HTML and some missing for JSON. For example, a lot of the code in the Form class is for handling hidden or disabled fields and generating templates, none of which is relevant in an API context. It is also missing support for schema generation.

However, there are many powerful concepts there for rapid development, especially when you consider the ModelForm. So my thought is, what if we broke down the core concepts of these forms to see how they could work as the interface for working with a more API-focused “mapper” interface (purposefully avoiding the terms “serializer” and “schema” from DRF and Ninja for clarity)? This interface could give a clean experience for mapping data from and to JSON-friendly objects. We could also create a ModelMapper that could be implemented by any sort of backend (forms, pydantic, cattrs, msgspec, etc.) with sufficient capabilities to map a dictionary of data into the shape of models, like ModelForm does now. I sketched this out a bit in a branch to explore interfaces. It is by no means a polished or complete interface. I created examples of what it could look like in function based views using either forms or pydantic/ninja schemas as the serialization backend. Notably, the idea is that while the instance backing logic is different, the developer experience stays the same. A slick experience would probably not require defining a class for the backend (form/schema) and for the mapper. Right now, though, I stuck with composition because it is easy to implement and read.

The second thought I had reviewing the CBV proposals is that the single form definition makes it feel tightly linked to how the HTML request/response flows work. In a HTML request/response flow, having a single form makes sense because once the form is submitted, you will usually redirect to a different view which will handle the response side of loading the data. In an API request/response flow, however, you won’t have a redirect, and you often rely on the response including the created resource in some form. Also, it is common for the structure of the incoming data to not match the structure of the outgoing data. For example, the incoming data for creating a resource called Employee linked to a Department may look like

{
  "name": "Chris M",
  "department_id": 1
}

However, the outgoing response might replace department_id for a hydrated department

{
  "name": "Chris M",
  "department": {
    "id": 1,
    "name": "Engineering"
  }
}

For this reason, I think a weakness in the current CBV interface is this single Form instance. My experiences building APIs has lead me to prefer the Ninja pattern of having one serializer/schema for in the input and another for the output. You can use the same for both, but it is common for me not to. It could be worth considering how we can update these interfaces to more natively support this separation of concerns.

4 Likes
47

+1 on Highlighting and supporting different serializers for input and output data. It doesn’t come up often but it littered throughout most API-based projects to be feel annoying.

Maybe Django can take a page from it’s own django.tasks story and have a generalised pluggable Mapper that third-party libs can override with a mixin or just support with some sort of plug and play in settings

CBVs seem to have evolved around the form and conventions of the pre-React world. For what it’s worth, I would like to add that I tend to prefer using DRF’s GenericMixin to logically group services and functions related to a service under the same class for discoverability and locality. Any CRUD operations are leveraged with Mixins as a happy bonus.

I would like to imagine a way to define serializers at the service level (maybe a decorator?) with a fallback within the CBV

48

To handle this, django-formset introduces Form-Collections. With these collections you can create deeply nested ModelForms in order to map foreign key relations. Here is a demo on how that works: 15. Creating Collections from related Modelsdjango-formset

49

Hi everyone,

I’d like to share some ideas from working with Django and DRF, especially around how Django could support async-first REST APIs natively.

After years of using DRF, I notice the basic workflow for newcomers is always:

  1. Define models
  2. Create serializers
  3. Add views/viewsets
  4. Register routers

It works, but often feels repetitive — models and serializers frequently duplicate the same fields.
So I’ve been thinking: what if, for most cases, Django could generate serializers automatically from models, only requiring custom serializers for special cases?

Model-Driven Serialization

Models could define API behavior through field attributes or Meta options:

class User(models.Model):
    email = models.EmailField(read_only=True)
    password = models.CharField(write_only=True)

    class Meta:
        api = {
            "exclude": ["is_staff"],
            "exclude_on": {"GET": ["password"]},
            "include_on": {"POST": ["password"]},
        }

From this, Django could build a default serializer automatically.
Developers could still define custom serializers when needed.
Under the hood, Django might include a base abstract serializer class, allowing other libraries (e.g., msgspec) to plug in as alternative backends via settings.

Simplified View Layer

Like DRF’s ModelViewSet, but with simpler serializer mapping:

class UserViewSet(APIViewSet):
    model = User
    serializers = {
        "get": UserListSerializer,
        "post": UserCreateSerializer,
        "promote": PromoteUserSerializer,
    }

    @action(detail=True, methods=["post"], serializer=PromoteUserSerializer)
    def promote(self, request, pk=None):
        ...

Resolution order could be:

  1. Serializer defined in @action
  2. Serializer from the mapping
  3. Explicit serializer_class
  4. Auto-generated serializer from model metadata

This keeps things declarative, schema-friendly, and compatible with tools like Swagger or Scalar.

Async-First and Django-Style

I hope this new API layer could be async-first by design.
Currently, DRF and Celery are the main blockers for full async in Django.
Even if sync remains for compatibility, async should be the default for performance.

Personally, I prefer class-based views over function-based dependency injection (as in FastAPI/Ninja).
CBVs make it easier to group authentication, permission, and logic together — more readable and consistent with the “Django way”.

In short:

  • Keep DRF’s strengths (ViewSets, mixins, permissions, routers)
  • Reduce boilerplate via model-driven defaults
  • Async-first design
  • Pluggable serializer backends
  • Stay aligned with Django’s CBV structure

How do you feel about this direction, and do you think it could fit well with Django’s ORM and async stack?

50

i personally don’t agree
i think keeping models and serializers serparate is a good thing
even if this was to happen, i think it should be a 3rd party

also, i don’t understand why you say celery is a blocker for async code

3 Likes
51
52

I want to keep moving this forward, and I think there are a number of building blocks that could help. Two of these have seen some discussion either in the new features repository or as a DEP already

  1. Path/Method routing: Http method routing · Issue #87 · django/new-features · GitHub
  2. JSON Parsing: https://github.com/django/deps/pull/88
  3. Backend-agnostic API mapping/validation/serialization (discussed here)

I think these building blocks make it possible to then consider the implementation of broader concepts that create an opinionated “framework” for building APIs

  • View grouping
  • Error handling and rendering
  • Multiple APIs on the same project
  • API middleware/decorating
  • Documentation/Schema generation
  • API-first CBV implementations
  • FBV shortcuts and decorators
  • Model-based dynamic API generation

I don’t believe Django needs to provide every part of this fully opinionated “framework” implementation, but I think some of these parts would live more naturally within Django. For example, I think error handling can best be handled at the Django level, since there it can also account for middleware/initialization/etc before the request even makes it to the API layer.

My proposal: I think that pushing the first three building blocks forward are good next steps to making progress on zags original concept “Add direct REST API support to Django”

My challenge: I’m not sure how to best help at this phase :grin: .

It would be great to get confirmation from those within the Django organization that help decide this path forward (steering council?) to avoid putting time to work that has no future. I’m not sure if that confirmation comes in this forum or would be better suited elsewhere (new feature repository/dep/etc).

Knowing the next actionable items for each of these initiatives would be step two. Does the existing parsing DEP need support? Would DEPs be appropriate for method routing or mapping? Are there other places to discuss at a technical level? Does this concept deserve something like a working group to help track it holistically? Understanding if and where I can best help would be awesome if anyone can help guide.

53

Some more inspiration for what is possible with Django and API-first development from the ecosystem.

Django-bolt is on the opinionated side and brings msgspec in for validation/mapping. It goes much farther and brings Rust server capabilities into play to aim for the absolute fastest API server experience as well, though.

Django-rapid is more inline with discussions here on how a mapping/validation layer could exist within Django using something like msgspec.

54

I was thinking along similar lines in terms of progress. I think it would be the Steering Council to set the direction here (but it has already started with the DEP referenced being the likely first candidate)

As for an immediate next step it’s probably picking up the DEP and getting a small group of people to push it forward.

@steering_council would you agree or have other discussions happened?

1 Like
55

The content type DEP would be the first thing to progress IMO. (It’s been the missing piece for a number of years now, like ≈ a decade :slight_smile: )

6 Likes