Settings refactor

Django Internals
1

At @apollo13 's suggestion I’m starting a topic here to hopefully consolidate discussion of the settings refactor that has come up on a few different threads, specifically:

There are also two separate but related issues:

I won’t repeat everything verbatim (read the threads for more detail), but at a high level, @apollo13 suggested:

I’d like to have a proper answer for handling settings rather then go setting by setting. The main difficulty I see is to decide whether we really go low-level in the sense of providing a simple API that can get settings from a variety of sources (the get("key", default)) or if we go further and provide a newer opinionated high level API like pydantic etc… Both approaches have their ups and downs.

What do folks think about how we might tackle this issue? Is there enough definition around it to create an issue in Trac at this point?

1 Like
2

Thanks for opening this @Tobias!

I’m just gonna lay down my rant on this, as an initial response. :stuck_out_tongue_winking_eye:

Is there enough definition around it to create an issue in Trac at this point?

Not normally what I’d say but, I think we actually need a DEP in this case.

  1. There are a number of settings solutions out there in the community — django-configurations, Django-environ, goodconf, envparse, django-classy-settings, … — We should be cognisant of what’s out there, and what each is doing well when coming up with a solution, so a review would be the first step. (This doesn’t necessarily need to be formal, but it should feed into what comes next.)
  2. Then, there’s a whole new world of, let’s call them, serialisation options: Pydantic and the pair of attrs/cattrs are the main libraries I think, but there’s the whole typing and dataclasses thing to look at , and decide OK, what do we in Django want to do? — Is it pick a library and take on a dependency, or do we implement something here ourselves?

I don’t want to make 2 there too big, such that we never achieve anything, but the exact same review of serialisation is needed in relation to whether we can do (JSON&co) APIs in core (which is something @andrewgodwin mentioned as a target project at DjangoCon). Neither the forms nor the serialisers that we have currently are suitable for this. (There’s a reason DRF exists in the first place.)

I think an answer to the question of what (de)serialization in Django is going to look like for the next decade is actually the project here. Then we use that for settings. (This entails a “let’s go high-level” answer to the question @apollo13 posted in the quote you have there.)

One project I think is very exciting in this space is Django-readers: https://www.django-readers.org — it let’s you define a spec for a specific nested structure, and then maps that back to (close to) the most efficient set of ORM queries (using only, and prefetches, and so on) that fetch exactly, and only, the requested data from the DB. I think it’s a really exciting project.

Why wouldn’t we just use Pydantic? — as a question that would clearly be raised. It’s because we want (and have the chance to get) this kind of deep integration into the ORM if we do this right. Leveraging what Pydantic and attrs/cattrs have learnt — maybe even building on either of them — but doing it with Django in mind has to be the goal.

So, then, if we have a answer to serialisation in place — a new newforms if you will :stuck_out_tongue_winking_eye: — we should use that for settings. With mappers from environment, from secret stores, from KDL files, etc. — perhaps just the protocol interfaces and some common cases there needed in core.

This project is for me Part 2 of Let’s get DRF into core: the first part is the work on content negotiation that I’m breaking ground on now. (The third would be generic views, but that would begin outside core, and maybe stay there, who knows.) My plan was to swing to this serialisation question in 2023. I’d very much like working with others on that. (In the meantime I’d likely keep using one of the existing options.)

END RANT. :melting_face:

4 Likes
3

I was thinking about this again last night.

I’m not particularly anti the lower-level approach. I think that’s what’s been initially suggested each time (and it’s a few times around the playground now) that the topic has come up.

The thought I had was, why haven’t these gone anywhere? — because the existing ecosystem solutions are good enough, and the difference with (say) just pulling env vars in the default template is it’s not enough of an improvement to merit the disruption.

Now is better than never

If we can’t make progress with the high-level approach (during say the 5.x cycle) we should maybe though do something rather than freeze forever. :thinking:

My only concern with Right now would be churn making a supposedly minimal change now only to swap it out basically immediately if we can progress the high-level story.

IDK :woman_shrugging: but that was the thought :slight_smile: