Settings refactor

Django Internals
1

At @apollo13 's suggestion I’m starting a topic here to hopefully consolidate discussion of the settings refactor that has come up on a few different threads, specifically:

There are also two separate but related issues:

I won’t repeat everything verbatim (read the threads for more detail), but at a high level, @apollo13 suggested:

I’d like to have a proper answer for handling settings rather then go setting by setting. The main difficulty I see is to decide whether we really go low-level in the sense of providing a simple API that can get settings from a variety of sources (the get("key", default)) or if we go further and provide a newer opinionated high level API like pydantic etc… Both approaches have their ups and downs.

What do folks think about how we might tackle this issue? Is there enough definition around it to create an issue in Trac at this point?

2 Likes
2

Thanks for opening this @Tobias!

I’m just gonna lay down my rant on this, as an initial response. :stuck_out_tongue_winking_eye:

Is there enough definition around it to create an issue in Trac at this point?

Not normally what I’d say but, I think we actually need a DEP in this case.

  1. There are a number of settings solutions out there in the community — django-configurations, Django-environ, goodconf, envparse, django-classy-settings, … — We should be cognisant of what’s out there, and what each is doing well when coming up with a solution, so a review would be the first step. (This doesn’t necessarily need to be formal, but it should feed into what comes next.)
  2. Then, there’s a whole new world of, let’s call them, serialisation options: Pydantic and the pair of attrs/cattrs are the main libraries I think, but there’s the whole typing and dataclasses thing to look at , and decide OK, what do we in Django want to do? — Is it pick a library and take on a dependency, or do we implement something here ourselves?

I don’t want to make 2 there too big, such that we never achieve anything, but the exact same review of serialisation is needed in relation to whether we can do (JSON&co) APIs in core (which is something @andrewgodwin mentioned as a target project at DjangoCon). Neither the forms nor the serialisers that we have currently are suitable for this. (There’s a reason DRF exists in the first place.)

I think an answer to the question of what (de)serialization in Django is going to look like for the next decade is actually the project here. Then we use that for settings. (This entails a “let’s go high-level” answer to the question @apollo13 posted in the quote you have there.)

One project I think is very exciting in this space is Django-readers: https://www.django-readers.org — it let’s you define a spec for a specific nested structure, and then maps that back to (close to) the most efficient set of ORM queries (using only, and prefetches, and so on) that fetch exactly, and only, the requested data from the DB. I think it’s a really exciting project.

Why wouldn’t we just use Pydantic? — as a question that would clearly be raised. It’s because we want (and have the chance to get) this kind of deep integration into the ORM if we do this right. Leveraging what Pydantic and attrs/cattrs have learnt — maybe even building on either of them — but doing it with Django in mind has to be the goal.

So, then, if we have a answer to serialisation in place — a new newforms if you will :stuck_out_tongue_winking_eye: — we should use that for settings. With mappers from environment, from secret stores, from KDL files, etc. — perhaps just the protocol interfaces and some common cases there needed in core.

This project is for me Part 2 of Let’s get DRF into core: the first part is the work on content negotiation that I’m breaking ground on now. (The third would be generic views, but that would begin outside core, and maybe stay there, who knows.) My plan was to swing to this serialisation question in 2023. I’d very much like working with others on that. (In the meantime I’d likely keep using one of the existing options.)

END RANT. :melting_face:

4 Likes
3

I was thinking about this again last night.

I’m not particularly anti the lower-level approach. I think that’s what’s been initially suggested each time (and it’s a few times around the playground now) that the topic has come up.

The thought I had was, why haven’t these gone anywhere? — because the existing ecosystem solutions are good enough, and the difference with (say) just pulling env vars in the default template is it’s not enough of an improvement to merit the disruption.

Now is better than never

If we can’t make progress with the high-level approach (during say the 5.x cycle) we should maybe though do something rather than freeze forever. :thinking:

My only concern with Right now would be churn making a supposedly minimal change now only to swap it out basically immediately if we can progress the high-level story.

IDK :woman_shrugging: but that was the thought :slight_smile:

4

I’m wondering what the smallest thing we could do here would be.

There is already a precedent with DJANGO_SETTINGS_MODULE of loading environment information from environment variables.

What about a backend that formalizes that and handles basic type casting (int, bool, string, list) such that it can be used in settings.py?

And make it so third parties can write their own backends to load from files or Hashicorp Vault or whatever.

Pydantic (and goodconf) are great, but I see too much friction trying to get them adopted for this use case.

With that in mind, I think django-environ comes closest and could get slimmed down quite a bit for a v1.

Getting ahead of myself, but one feature from goodconf I’d love to see (which django-environ also supports) is an upfront definition of the variables that can be loaded. Scattering them throughout the project makes it hard to know options are available. Having them all defined in one place opens the door to being able to auto-generate documentation.

5

@ipmb Thanks for swinging back to this! I think something here would be appropriate.

I quite the way Hynek’s take on this has compatibility with various kinds of secrets stores. It would be good to define an API (in the Django way) and provide probably the env vars backend for that, which could be extended by folks wanting systemd credentials, Vault storage, …, and you name it.

In saying that, I don’t want to stretch the initial goals too far, but the big objection that came up every time this was discussed, and is probably correct, is that many teams won’t go near env vars, for reasons.

6

I get the hate for environment variables, but the fact is that’s how every PaaS (that I know of) does it. I’ve largely given up on that battle. Whatever we choose won’t please everyone and that’s fine. We should start somewhere and despite their faults, environment variables are the most common way this is done.

I’m 100% in agreement on providing an API that can be extended. Follow the pattern of databases, storages, etc.

environ-config seems fine, but it has an additional dependency which I assume folks will also be concerned about.

Personally, I’d love to see type hints used for casting, but that’s probably going too far as well :slight_smile:. Probably more important to answer “what?” than “how?” at this point.

2 Likes
7

Agreed. I think all we’d need is the interface and then the single env var backend. Happy to review any draft you come up with.

(It’s time we did this.)

1 Like
8

Thanks for picking this up @ipmb . I’d also be interesetd in reviewing any draft.

1 Like
9

Here’s a rough strawman to start the discussion Django Configuration Loader Strawman · GitHub