Like many others, I disagreed with the decision. But we have a board, and they’re in charge. You can’t please everyone.
I feel like it’s time to revisit this. For one, the typing community has made enormous progress and django-stubs is a godsend for my codebases. Mypy is more expressive as well.
But I realize this is a big decision and maybe Django is still not ready. Instead, I’m proposing guidance on new code. Specifically, DEP-14 adding background tasks.
And it’s fully typed, includingenqueue with dynamic args. This itself is a small but valuable benefit over some other libraries. In fact, typing could even forbid non-serializable args using a JSON type and things like that.
It’d be a shame to manually and forcefully strip the resulting PR of its types. Could we consider leaving them in?
I think there’s broadly these options for Django and typing at this point:
Continue to forbid typing in Django core.
Require typing for new and old features.
Require typing for new features.
Allow typing for new features (up to the author).
I am still personally not really a fan of typing - I don’t get much benefit out of it personally and fairly often find myself fighting the type checker (specifically mypy). That said, I know a lot of people get value out of it and the tradeoffs are different for a library than for an application. Typing of a library is an extra feature its users can choose to take advantage of.
For these reasons, I’d be opposed to my options 2 and 3 at this point. But I think typing has grown enough that I don’t think option 1 should automatically win now. I think Django tasks is a good motivating use case to choose option 4.
I welcome further discussion here and I think once we’ve had that discussion, a DEP would be the right next step. Once there’s a DEP, the Steering Council will review and make a decision.
I do not think that would be very useful. Consider a function (new feature) taking HttpRequest as an argument. Without having a fully annotated HttpRequest then the typing options you have inside that function are kinda limited, no?
I think django-stubs would continue to be a requirement for the foreseeable future for any code that isn’t typed in Django itself. This would essentially reduce the burden for the django-stubs maintainers for some new features.
I agree, it has been some extra work stripping the types from django-tasks when opening the Django PR, but in the grand scheme of things it’s really not that much.
Making an API type safe doesn’t necessarily mean there need to be type hints. django-types was developed to be type-safe, meaning it behaves predictably with its types. That’s a value even without hints.
As for getting types in Django, it’s a hard problem to solve. It’s an interesting discussion, but I don’t think it’s one which necessarily needs solving for DEP 14. It’s potentially high impact given the size of the Django codebase and community and probably shouldn’t be taken lightly. It’s worthy of its own dedicated discussion at some point (I’m looking at you, SC).
As a user I’d really like types. I’d especially like types that work fully with pyright as it works without plugins and doesn’t need to run any code, and I find it a better piece of software than mypy in particular. However, django-stubs has only recently began supporting anything other than mypy and the level of support since then has been quite lacking IMO.
As a contributor though, it’s quite scary. A lot of typing syntax (necessarily, because of the theory that goes into it) is complex. Now I need to understand covariance, invariance, and contra-variance, which, even with a CS background, I don’t. No doubt I could learn it, but it’s yet another thing I need to keep in my head when writing a feature or maintaining some code somewhere.
Sure, it’d be great to have. I just don’t know if the benefits outweigh the costs. If Django was a new project, I think it would have types. However, it isn’t, and there are a lot of very dynamic parts that in my understanding require some advanced techniques to type correctly.
I wouldn’t stand in the way of it, but I’m also not in any rush to have it.
This is one of the major points from the older discussion. We already see many points about how difficult it is to get a patch into Django. I have some thoughts here but it’s something we need to keep in mind.
I don’t think this needs to be an instant discussion. Lots of points to think through. I’m pleased to see it started.
I personally have less than zero interest in running a static type checker on my codebase, in large part due to having spent too much time on pointless fights with mypy.
However, I find incredible value in the runtime features type hints can enable and that they have enabled in other libraries/frameworks. SQLAlchemy since 2.0, for example, can derive the base DB schema from type hints on your model class’ attributes, and other libraries can derive input/output serialization and validation from them, web frameworks can derive expected URL parameters, querystring, request payload, etc.
I would love to open up the world of those features for Django, but doing so would require making type hints a first-class thing in Django rather than an optional and officially-unsupported and officially-discouraged add-on you have to go hunt around and use third-party libraries for.
Also, IIRC there was at one point funding on the table to support a project of type-hinting all of Django, but that was years ago and I imagine whoever put up the money got frustrated and gave up.
I agree with what’s been pointed out re: user interest in stubs and the runtime features of type annotations, but I also think we shouldn’t overlook the benefits of static typing for the development of Django itself.
The ORM in particular is an area that I think would benefit from type annotations. We want to onboard more contributors in this area. Simon has an excellent walkthrough video, and the functions are named intuitively, but type annotations would assist folks as they poke around.
Perhaps more importantly, I’m convinced we will discover bugs as we do this work. ticket-35972 could have been caught by a static type checker. I just put up a PR, but I’m almost certain there are more failing cases I don’t know about–cases we won’t find until ensure the ORM is type-correct.
Type-annotating the ORM will also make it more friendly for developers to override. I joined a project that was overriding get_prep_value(), but when I found cases that were missed, I had to quickly get my arms around whether what we were doing was even supported at all. I filed a ticket, and Simon helpfully explained that in the absence of type annotations, the gray area rounded down to not supported, but that with type annotations there might be another story someday.
All this to say: I think we can decide component by component how to proceed here. For me, the value proposition for type annotations is highest with the ORM.
To add to what @jacobtylerwalls said I think that very minimal typing around the introduction of two typing.Protocol could go a long way in making the ORM more approachable
from __future__ import annotations
from typing import Protocol
from django.db.models import fields
from django.db.models.sql import compiler, query
class Compilable(Protocol):
"""
Object that can be compiled to a PEP 249 compliant tuple comprised of
a SQL string and its associated parameters sequence.
It can optionally have `as_vendor` methods defined to specialize the
compilation on different backends (e.g. `as_postgres`).
"""
def get_source_expressions(self) -> list[Compilable | None]: ...
def as_sql(
self, compiler: compiler.SQLCompiler, connection
) -> tuple[str, tuple]: ...
class Resolvable(Protocol):
"""
Object that can resolve any field and transform reference it has in
the context of a query and return an object ready to be compiled for
execution.
"""
output_field: fields.Field
def get_source_expressions(self) -> list[Resolvable | None]: ...
def resolve_expression(
self,
query: query.Query,
allow_joins: bool = True,
reuse: set[str] | None = None,
summarize: bool = False,
for_save: bool = False,
) -> Compilable: ...
Typing adds some challenges when writing new code. However, I feel it makes it easier to maintain the code. When working on existing, unfamiliar code, typing helps a lot.
We’re not ready to require types for new code. But I do think we should accept work to add types, and as we do we should start to require them in the parts of the code where they are added, as a ratchet. This means that we also need to have a high amount of validation on the places that we add types.
Wrong types are worse than no types, so this is work that we’ll need to take carefully. I’m looking forward to this work finding bugs, which I very much expect. I think we’re also going to find places where our API designs just don’t play nice with the abilities of available type checkers, because they predated a lot of the recent typing work in Python and the patterns don’t align.
I think that it would even be wise to be open to breaking changes to allow for working types. Those choices will have to be made on a case-by-case basis, and we’ll have to weigh the risks, but I think we’ll find that if type checkers aren’t able to type some of our code patterns, that we should at least consider the possibility that they aren’t optimal patterns with the current state of Python, and we should find better patterns.
We’ll need to coordinate the work to add types. To figure out what the staged integration path is. And it’ll need cooperation and clear communication from trusted community members to validate the types. The fellows won’t be able to coordinate this project, so it will take someone willing and able to communicate the plan, how it should be validated, and any compromises that need to be made in the implementation, to minimize the work mergers need to do to validate that the target has been met.
There is another option here that I think is important and not listed, especially given the… inexact nature of even deeper plugins like django-stubs, which is “add types to things which are easy to add types to, and not treat typing as a deeper blocker”. The point here being that types should be attempted, but should not block progress.
I have been dealing with ORM internals and types would be quite helpful for a good subset of the code where there aren’t issues of dependent typing or other magical things that make some other parts of Django near-impossible.
Adding the right kind of type annotations is definitely a skill, just like API design in general. I definitely think that types that would be within Django proper should be right. And honestly, given the current state of mypy in particular, that precludes shipping types on everything.
But we can still provide clean types for a lot of functions. And for more tricky things, there are type signatures that would be good! But they would need to be desigend right. You need to choose the right kind of generic parameters to get something that doesn’t get in the way, and it might not be the first type signature someone comes up with.
There’s some stuff where type signatures would provide very valuable documentation, even if the type checking component would be limited. Things like genericizing Field to actually have an indication of what python value is being returned (is IPAdressField returning a bespoke object or just a string?)
The biggest problem with only typing some of the codebase is then people might treat untyped stuff as an upstream bug or “missed spot”.
I just want to add that I have noticed a disconnect between the type checker often used by package maintainers (mypy) and general python developers (pylance/pyright and pycharm).
The most actively developed stubs for Django is django-stubs and depends on mypy but I would wager that the overwhelming majority of Django/Python users are not using mypy.
django-types is a noble attempt to remove the mypy dependency, making it more compatible with the remainder of the ecosystem and the larger user base (if my observations are representative). This is however hard without support from Django itself.
This discussion might mostly be scoped to the internals of Django but I think the developer experience is arguably more important and it mostly depends on the public/documented interface being type annotated.
Type hints in editors reduces the friction of writing code and is a significant boon to newer users of the framework. It guides them on a path with fewer obstacles, reducing the number of trial and error cycles, maintaining their momentum and morale. I believe this will also improve the contributor experience.
I hope that the recent Django survey will shed light on the usage statistics to bring some concrete numbers to the discussion.
Could someone clarify the difference between runtime and static typing in this context? I get the general idea but don’t quite see how it applies to this discussion. As I understand it, this is about adding type hints to function parameters and return values.
For what it’s worth, I’m very much in favor of this. I’ve been using Python since 2008 (not professionally) and understand the duck typing philosophy. But typing introduces an interesting wrinkle. If something looks and behaves like a duck, it’s a duck.
The reverse is also true: to prove that something is a duck, you have to call quack() and hope it works, which isn’t exactly ideal. Typing doesn’t take away Python’s flexibility, but it does make intent clearer. And while it’s not perfect—IntelliSense might still suggest quack() on the wrong kind of object—it can eliminate a lot of guesswork. My main frustration is when PyCharm fails to infer the type of a Django object, leaving me to dig through documentation just to figure out what I’m dealing with. That alone makes typing worth it.
As for concerns about added complexity for Django developers—I get that this project depends on volunteers. But user demand is what keeps an open-source project alive, and typing isn’t going away. It’s an active part of Python’s evolution, and Django should engage with it sooner rather than later.
Here’s a quick proposal from someone who checks their code with both mypy and pyright. (I’ve been using django-types for the last 3 years to avoid needing to deal with the mypy plugin, and don’t think pyright works with django-stubs.)
Don’t require types, but allow people to add typing.
Don’t require everyone to understand types. If typing is getting in the way for a pull request or new feature, add # type: ignore wherever needed, and let the people who care about types come along and improve things later (ideally before the alpha cut).
Add mypy to continuous-integration and require it to pass, but again, it’s fine to leave things untyped or use # type: ignore to get it to pass.
Work with the django-stubs and/or django-types projects to merge those types into the project as a way to start. This also helps the types stay in sync with the right Django version. Leave the mypy plugin as a separate project because it’s useful, but probably too complicated to merge into Django.
If maintaining a node enviroment in the continuous-integration isn’t an issue, consider also adding a pyright check in addition to the mypy check.
The main downside is the code is more verbose, especially for those not used to working in a typed codebase.
I can’t speak for what anyone else means, but personally what I mean is that there are a lot of tools and libraries and frameworks now which use type hints to determine runtime behavior. For example, in the SQLAlchemy ORM you can now write a model class like this:
from datetime import date
from sqlalchemy.orm import DeclarativeBase, Mapped
Base = DeclarativeBase()
class Person(Base):
__tablename__ = "person"
name: Mapped[str]
date_of_birth: Mapped[date]
And while it’s true that a static type checker could now flag code that tries to, say, assign an int to the name field, it’s also true that this is all the declaration SQLAlchemy needs to emit the correct table-creation and other SQL and transform values going to/from the database at runtime, because it can inspect the type hints to figure out the correct column types.
Other libraries like Pydantic and msgspec use type hints to derive runtime validation, serialization and de-serialization behavior. Web frameworks like FastAPI and Litestar use type hints on HTTP route-handler functions to specify at runtime the acceptable/required input and output formats. For example, in Litestar if you want to declare that an HTTP route handler has a required querystring parameter page_number, you could write:
from litestar import get
@get("/path")
async def my_endpoint(page_number: int) -> dict:
return {"requested_page": page_number}
And Litestar would, with no further code needed, enforce the presence of that querystring parameter, including adding it to generated OpenAPI schemas, returning an appropriate automatic error response if you make a request without the parameter, etc. FastAPI and Litestar also use type hints to drive dependency injection and other functionality.
These are all powerful use cases for type hints at runtime, and are mostly decoupled from trying to statically check the “correctness” of a program. But they only work if you have the type hints in the first place.