Thanks for opening this @Tobias!
I’m just gonna lay down my rant on this, as an initial response.
Is there enough definition around it to create an issue in Trac at this point?
Not normally what I’d say but, I think we actually need a DEP in this case.
- There are a number of settings solutions out there in the community — django-configurations, Django-environ, goodconf, envparse, django-classy-settings, … — We should be cognisant of what’s out there, and what each is doing well when coming up with a solution, so a review would be the first step. (This doesn’t necessarily need to be formal, but it should feed into what comes next.)
- Then, there’s a whole new world of, let’s call them, serialisation options: Pydantic and the pair of
cattrs are the main libraries I think, but there’s the whole typing and dataclasses thing to look at , and decide OK, what do we in Django want to do? — Is it pick a library and take on a dependency, or do we implement something here ourselves?
I don’t want to make 2 there too big, such that we never achieve anything, but the exact same review of serialisation is needed in relation to whether we can do (JSON&co) APIs in core (which is something @andrewgodwin mentioned as a target project at DjangoCon). Neither the forms nor the serialisers that we have currently are suitable for this. (There’s a reason DRF exists in the first place.)
I think an answer to the question of what (de)serialization in Django is going to look like for the next decade is actually the project here. Then we use that for settings. (This entails a “let’s go high-level” answer to the question @apollo13 posted in the quote you have there.)
One project I think is very exciting in this space is Django-readers: https://www.django-readers.org — it let’s you define a
spec for a specific nested structure, and then maps that back to (close to) the most efficient set of ORM queries (using
only, and prefetches, and so on) that fetch exactly, and only, the requested data from the DB. I think it’s a really exciting project.
Why wouldn’t we just use Pydantic? — as a question that would clearly be raised. It’s because we want (and have the chance to get) this kind of deep integration into the ORM if we do this right. Leveraging what Pydantic and attrs/cattrs have learnt — maybe even building on either of them — but doing it with Django in mind has to be the goal.
So, then, if we have a answer to serialisation in place — a new newforms if you will — we should use that for settings. With mappers from environment, from secret stores, from KDL files, etc. — perhaps just the protocol interfaces and some common cases there needed in core.
This project is for me Part 2 of Let’s get DRF into core: the first part is the work on content negotiation that I’m breaking ground on now. (The third would be generic views, but that would begin outside core, and maybe stay there, who knows.) My plan was to swing to this serialisation question in 2023. I’d very much like working with others on that. (In the meantime I’d likely keep using one of the existing options.)