Document StorageHandler as public API.

Django Internals
1

Django 4.2 introduced the new storages API in order to allow folks to define and use multiple storages.

Internally storages is an instance of the StorageHandler class.

I would like to minimally document that, in order to officially make it available to third-party package maintainers.

Packages need to provide storage instances, and in order to opt-in to the new storages API need to be able to allow users to provide a storage for a known alias (the name of which is most easily given by a setting). If the user provides such a storage that should be used, otherwise the third-party package should provide a default.

That would look something like this:

alias = # From e.g. settings
storage_class = # Default class to use if not overridden. 
try:
    # Did the user provide the alias? 
    return storages[alias]
except InvalidStorageError:
    # Update storages to provide a default. 
    storages.backends[alias] = {
        "BACKEND": storage_class
    }
    return storages[alias]

This works perfectly, but, in order to be safe, the backends dict should be documented.

The create_storage() method is also useful, and should be made available.

FWIW this comes up Updating django-compressor to use the new API but the same would be necessary for any third-party package wanting to allow users to use the storages API in order to provide optional configuration overrides.

1 Like
2

Iā€™ve created a PR so you can see what I have in mind.

3

+1 to documenting storages. I envisage it to be similar to the documentation for caches.

Also, I found that django.db.connections does not have proper reference documentation, just some usage coverage in Using raw cursors with multiple databases and Executing custom SQL directly. I think it deserves a similar section to caches.

I am not so hot on encouraging the pattern of inserting backends into the handler class. Why canā€™t the package require users to define the key in STORAGES? That seems more transparent to me, and more aligned with the other ā€œhandlersā€ (caches / databases).

4

Thatā€™s a good question. The answer is that 99% of users are never going to customise the storage here, theyā€™re just going to use the default. So itā€™s adding a quite a big extra boilerplatey-horrible-pointlessness to make them add the STORAGES entry.

The pattern we want is ā€œDid they override?ā€ If so use, otherwise provide the default. We could not add the storage to storages but then next call we canā€™t easily re-use the instance.

Adding it to the backends dict, and then accessing it works just nicely and resolves all these issues.

Iā€™m still open to another way of letting folks opt-in to the storages API with third-party packages providing the default, if there are any (but this have been bubbling on the compressor repo since 4.2, and this is the least touch solution thatā€™s presented so far).

All I really want with the PR here is the stability guarantee that we can use the StorageHandler as implemented without a non-deprecation path breaking change. (i.e. We could just mention backends and not have the example, for example.)

Update: FWIW I was thinking about an alternate for compressor not updating backends, for which weā€™d need to instantiate a storage ourselves. That would work, but weā€™d loose the caching. For that though weā€™d need the create_storage() method, or weā€™d have to vendor that, with the complications on keeping it up to date that that would entail, so it would still be better to have it part of Djangoā€™s public API.

5

@adamchainz @nessita @sarahboyce Do you think we can advance this? I donā€™t think weā€™re going to get more input.

As an extra data point, Iā€™ve helped a couple of people on separate projects, who hit the underlying issue here, and needed the StorageHandler API. (I asked both to comment here, but that hasnā€™t happened :pensive:)

6

Hey @carltongibson, I have been following along this thread and thinking about it.
Iā€™m currently in that awkward place where the proposed solution does not feel ā€œrightā€ but I donā€™t have a better counterproposal to offer, so I feel somehow blocked.

There are a few storage-related improvements pending, such as revisiting the API which is a bit confusing at times, specifically the file name and file path validation. Perhaps we could plan for brainstorm session to consider an API change that would most ā€œofficiallyā€ support this requirement?

7

Hey @nessitaā€¦ Yes, the storages API. Thatā€™s a bigger discussion :sweat_smile: We can chat at DjangoCon (though I canā€™t promise I have any great thoughts on itā€¦ ā€” youā€™re well ahead of me, having been in there recently.)

As I read @adamchainzā€™s reply he was +1 but with the query about the example. So, the question isā€¦

i.e. We could just mention backends and not have the example, for example.

Iā€™m totally happy to drop the example if itā€™s a bit shifty :supervillain:

@adamchainz also wanted to document django.db.connections ā€” Iā€™d be +1 for that, but it seems like a separate Ticket and PR.

WDYT?