Like many others, I disagreed with the decision. But we have a board, and they’re in charge. You can’t please everyone.
I feel like it’s time to revisit this. For one, the typing community has made enormous progress and django-stubs is a godsend for my codebases. Mypy is more expressive as well.
But I realize this is a big decision and maybe Django is still not ready. Instead, I’m proposing guidance on new code. Specifically, DEP-14 adding background tasks.
And it’s fully typed, includingenqueue with dynamic args. This itself is a small but valuable benefit over some other libraries. In fact, typing could even forbid non-serializable args using a JSON type and things like that.
It’d be a shame to manually and forcefully strip the resulting PR of its types. Could we consider leaving them in?
I think there’s broadly these options for Django and typing at this point:
Continue to forbid typing in Django core.
Require typing for new and old features.
Require typing for new features.
Allow typing for new features (up to the author).
I am still personally not really a fan of typing - I don’t get much benefit out of it personally and fairly often find myself fighting the type checker (specifically mypy). That said, I know a lot of people get value out of it and the tradeoffs are different for a library than for an application. Typing of a library is an extra feature its users can choose to take advantage of.
For these reasons, I’d be opposed to my options 2 and 3 at this point. But I think typing has grown enough that I don’t think option 1 should automatically win now. I think Django tasks is a good motivating use case to choose option 4.
I welcome further discussion here and I think once we’ve had that discussion, a DEP would be the right next step. Once there’s a DEP, the Steering Council will review and make a decision.
I do not think that would be very useful. Consider a function (new feature) taking HttpRequest as an argument. Without having a fully annotated HttpRequest then the typing options you have inside that function are kinda limited, no?
I think django-stubs would continue to be a requirement for the foreseeable future for any code that isn’t typed in Django itself. This would essentially reduce the burden for the django-stubs maintainers for some new features.
I agree, it has been some extra work stripping the types from django-tasks when opening the Django PR, but in the grand scheme of things it’s really not that much.
Making an API type safe doesn’t necessarily mean there need to be type hints. django-types was developed to be type-safe, meaning it behaves predictably with its types. That’s a value even without hints.
As for getting types in Django, it’s a hard problem to solve. It’s an interesting discussion, but I don’t think it’s one which necessarily needs solving for DEP 14. It’s potentially high impact given the size of the Django codebase and community and probably shouldn’t be taken lightly. It’s worthy of its own dedicated discussion at some point (I’m looking at you, SC).
As a user I’d really like types. I’d especially like types that work fully with pyright as it works without plugins and doesn’t need to run any code, and I find it a better piece of software than mypy in particular. However, django-stubs has only recently began supporting anything other than mypy and the level of support since then has been quite lacking IMO.
As a contributor though, it’s quite scary. A lot of typing syntax (necessarily, because of the theory that goes into it) is complex. Now I need to understand covariance, invariance, and contra-variance, which, even with a CS background, I don’t. No doubt I could learn it, but it’s yet another thing I need to keep in my head when writing a feature or maintaining some code somewhere.
Sure, it’d be great to have. I just don’t know if the benefits outweigh the costs. If Django was a new project, I think it would have types. However, it isn’t, and there are a lot of very dynamic parts that in my understanding require some advanced techniques to type correctly.
I wouldn’t stand in the way of it, but I’m also not in any rush to have it.
This is one of the major points from the older discussion. We already see many points about how difficult it is to get a patch into Django. I have some thoughts here but it’s something we need to keep in mind.
I don’t think this needs to be an instant discussion. Lots of points to think through. I’m pleased to see it started.
I personally have less than zero interest in running a static type checker on my codebase, in large part due to having spent too much time on pointless fights with mypy.
However, I find incredible value in the runtime features type hints can enable and that they have enabled in other libraries/frameworks. SQLAlchemy since 2.0, for example, can derive the base DB schema from type hints on your model class’ attributes, and other libraries can derive input/output serialization and validation from them, web frameworks can derive expected URL parameters, querystring, request payload, etc.
I would love to open up the world of those features for Django, but doing so would require making type hints a first-class thing in Django rather than an optional and officially-unsupported and officially-discouraged add-on you have to go hunt around and use third-party libraries for.
Also, IIRC there was at one point funding on the table to support a project of type-hinting all of Django, but that was years ago and I imagine whoever put up the money got frustrated and gave up.
I agree with what’s been pointed out re: user interest in stubs and the runtime features of type annotations, but I also think we shouldn’t overlook the benefits of static typing for the development of Django itself.
The ORM in particular is an area that I think would benefit from type annotations. We want to onboard more contributors in this area. Simon has an excellent walkthrough video, and the functions are named intuitively, but type annotations would assist folks as they poke around.
Perhaps more importantly, I’m convinced we will discover bugs as we do this work. ticket-35972 could have been caught by a static type checker. I just put up a PR, but I’m almost certain there are more failing cases I don’t know about–cases we won’t find until ensure the ORM is type-correct.
Type-annotating the ORM will also make it more friendly for developers to override. I joined a project that was overriding get_prep_value(), but when I found cases that were missed, I had to quickly get my arms around whether what we were doing was even supported at all. I filed a ticket, and Simon helpfully explained that in the absence of type annotations, the gray area rounded down to not supported, but that with type annotations there might be another story someday.
All this to say: I think we can decide component by component how to proceed here. For me, the value proposition for type annotations is highest with the ORM.
To add to what @jacobtylerwalls said I think that very minimal typing around the introduction of two typing.Protocol could go a long way in making the ORM more approachable
from __future__ import annotations
from typing import Protocol
from django.db.models import fields
from django.db.models.sql import compiler, query
class Compilable(Protocol):
"""
Object that can be compiled to a PEP 249 compliant tuple comprised of
a SQL string and its associated parameters sequence.
It can optionally have `as_vendor` methods defined to specialize the
compilation on different backends (e.g. `as_postgres`).
"""
def get_source_expressions(self) -> list[Compilable | None]: ...
def as_sql(
self, compiler: compiler.SQLCompiler, connection
) -> tuple[str, tuple]: ...
class Resolvable(Protocol):
"""
Object that can resolve any field and transform reference it has in
the context of a query and return an object ready to be compiled for
execution.
"""
output_field: fields.Field
def get_source_expressions(self) -> list[Resolvable | None]: ...
def resolve_expression(
self,
query: query.Query,
allow_joins: bool = True,
reuse: set[str] | None = None,
summarize: bool = False,
for_save: bool = False,
) -> Compilable: ...
Typing adds some challenges when writing new code. However, I feel it makes it easier to maintain the code. When working on existing, unfamiliar code, typing helps a lot.
We’re not ready to require types for new code. But I do think we should accept work to add types, and as we do we should start to require them in the parts of the code where they are added, as a ratchet. This means that we also need to have a high amount of validation on the places that we add types.
Wrong types are worse than no types, so this is work that we’ll need to take carefully. I’m looking forward to this work finding bugs, which I very much expect. I think we’re also going to find places where our API designs just don’t play nice with the abilities of available type checkers, because they predated a lot of the recent typing work in Python and the patterns don’t align.
I think that it would even be wise to be open to breaking changes to allow for working types. Those choices will have to be made on a case-by-case basis, and we’ll have to weigh the risks, but I think we’ll find that if type checkers aren’t able to type some of our code patterns, that we should at least consider the possibility that they aren’t optimal patterns with the current state of Python, and we should find better patterns.
We’ll need to coordinate the work to add types. To figure out what the staged integration path is. And it’ll need cooperation and clear communication from trusted community members to validate the types. The fellows won’t be able to coordinate this project, so it will take someone willing and able to communicate the plan, how it should be validated, and any compromises that need to be made in the implementation, to minimize the work mergers need to do to validate that the target has been met.
There is another option here that I think is important and not listed, especially given the… inexact nature of even deeper plugins like django-stubs, which is “add types to things which are easy to add types to, and not treat typing as a deeper blocker”. The point here being that types should be attempted, but should not block progress.
I have been dealing with ORM internals and types would be quite helpful for a good subset of the code where there aren’t issues of dependent typing or other magical things that make some other parts of Django near-impossible.
Adding the right kind of type annotations is definitely a skill, just like API design in general. I definitely think that types that would be within Django proper should be right. And honestly, given the current state of mypy in particular, that precludes shipping types on everything.
But we can still provide clean types for a lot of functions. And for more tricky things, there are type signatures that would be good! But they would need to be desigend right. You need to choose the right kind of generic parameters to get something that doesn’t get in the way, and it might not be the first type signature someone comes up with.
There’s some stuff where type signatures would provide very valuable documentation, even if the type checking component would be limited. Things like genericizing Field to actually have an indication of what python value is being returned (is IPAdressField returning a bespoke object or just a string?)
The biggest problem with only typing some of the codebase is then people might treat untyped stuff as an upstream bug or “missed spot”.