Storage 4.2: how to subclass default

[Swannbm]

Hello fellowes,

I am discovering new Django 4.2 storages settings. I want to have a main file system (default) then I would like to customise media folder on top of it (add media subfolder to all uploaded files and public-read ACL).

Before, I would have made a custom storage class inheriting of S3Boto… But it doesn’t work anymore (myabe I am doing something wrong).

I’ve endup doing below code but I don’t find that elegant.

Do you see another way to do it ?

DEFAULT_S3_STORAGE_OPTIONS = {
    "access_key": env.str("OVH_ACCESS_KEY"),
    "secret_key": env.str("OVH_SECRET_KEY"),
    "bucket_name": env.str("BUCKET_NAME"),
    "region_name": env.str("BUCKET_REGION"),
    "endpoint_url": "https://s3.gra.io.cloud.ovh.net",
    "custom_domain": f"{env.str('BUCKET_NAME')}.s3.gra.io.cloud.ovh.net",
}
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            **DEFAULT_S3_STORAGE_OPTIONS,
        },
    },
    "public_media": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            **DEFAULT_S3_STORAGE_OPTIONS,
            "location": "media",
            "default_acl": "public-read",
            "file_overwrite": True,
            "querystring_auth": False,
        },
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
    },
}

++
Swann

[leandrodesouzadev]

Hey there!
I don’t know if i got your question right, but i will show up how i configured public/private media on s3 with django-storages.

Versions used:

Django==4.2
django-storages==1.13.2

## my settings.py (partial)
# Storages
DEFAULT_FILE_STORAGE_BACKEND = env(
    "DEFAULT_FILE_STORAGE_BACKEND",
    default="django.core.files.storage.FileSystemStorage",
)
STORAGES = {
    "default": {"BACKEND": DEFAULT_FILE_STORAGE_BACKEND},
    "public_media": {"BACKEND": DEFAULT_FILE_STORAGE_BACKEND},
    "staticfiles": {"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage"},
}
AWS_ACCESS_KEY_ID = env.str("AWS_ACCESS_KEY_ID", default=None)
AWS_SECRET_ACCESS_KEY = env.str("AWS_SECRET_ACCESS_KEY", default=None)
AWS_REGION_NAME = env.str("AWS_REGION_NAME", default=None)

if DEFAULT_FILE_STORAGE_BACKEND == "app.ext.storage.aws_s3.PrivateMediaStorage":
    if AWS_ACCESS_KEY_ID is None or AWS_SECRET_ACCESS_KEY is None:
        raise ImproperlyConfigured("Missing AWS credentials")

    STORAGES["public_media"].update(BACKEND="app.ext.storage.aws_s3.PublicMediaStorage")
    INSTALLED_APPS.append("storages")
    AWS_STORAGE_BUCKET_NAME = env("AWS_STORAGE_BUCKET_NAME", str)
    AWS_S3_CUSTOM_DOMAIN = f"{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com"
    AWS_S3_OBJECT_PARAMETERS = {
        "CacheControl": "max-age=86400",
    }
    AWS_MEDIA_LOCATION = "media/"

And storages:

## app/ext/storage/aws_s3.py

from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage


class PrivateMediaStorage(S3Boto3Storage):
    """Stores privately on Aws S3"""

    location = getattr(settings, "AWS_MEDIA_LOCATION", "media")
    default_acl = "private"
    file_overwrite = False
    custom_domain = False


class PublicMediaStorage(S3Boto3Storage):
    """Stores public on Aws S3"""

    location = getattr(settings, "AWS_MEDIA_LOCATION", "media")
    default_acl = "public-read"
    file_overwrite = False
    custom_domain = False
    querystring_auth = False

The way it works for me is, set the default storage backend only, in my case i set it to use the private media by default, that being app.ext.storage.aws_s3.PrivateMediaStorage specifically. So when i need to use the public_media i use this function on the file/image fields:

## app/ext/storage/utils.py

from django.core.files.storage import storages


def get_public_media_storage():
    return storages["public_media"]

Like so:

class SomeModel(models.Model):
    picture = models.ImageField(
        verbose_name=_("Picture"),
        upload_to="events/events_pictures",
        storage=get_public_media_storage,
    )

[Swannbm]

Yes, thank you for replying leandrodesouzadev. It was also my way to do it before 4.2 and the STORAGES settings.

In documentation it says “A dictionary containing the settings for all storages to be used with Django.” which I emphasis as we shouldn’t have to create PrivateMediaStorage or PublicMediaStorage classes somewhere else. The idea, as far as I understand, is to get all storages in one place (in the settings, which sound very good).

Moreover, I think prefixed AWS_… settings should be avoided if possible. It’s better because there are plenty of other S3 providers nowadays and the naming could be confusing. This makes the STORAGES settings pretty good to me.

But I really don’t like not being DRY and I can’t see another way to do it than the solution I posted, which doesn’t look very good to me right now. But I don’t see how to do it differently…

I imaginated, instead of making new class, would be to reference a previously defined storage in the BACKEND option.

Example:

DEFAULT_S3_STORAGE_OPTIONS = {
    
}
STORAGES = {
    "default": {
        "BACKEND": "storages.backends.s3.S3Storage",
        "OPTIONS": {
            "access_key": env.str("OVH_ACCESS_KEY"),
            "secret_key": env.str("OVH_SECRET_KEY"),
            "bucket_name": env.str("BUCKET_NAME"),
            "region_name": env.str("BUCKET_REGION"),
            "endpoint_url": "https://s3.gra.io.cloud.ovh.net",
            "custom_domain": f"{env.str('BUCKET_NAME')}.s3.gra.io.cloud.ovh.net",
        },
    },
    "public_media": {
        "BACKEND": "default",  # reuse previously defined default with same options
        "OPTIONS": {
            # add or override options defined in default
            "location": "media",
            "default_acl": "public-read",
            "file_overwrite": True,
            "querystring_auth": False,
        },
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
    },
}

Do you think it would make sense ?
Or do you have some other idea ?

[leandrodesouzadev]

Well, if you to go that way, then i believe you can get around doing this:

default_storage_backend = "storages.backends.s3.S3Storage"
default_storage_options = {
    "access_key": env.str("OVH_ACCESS_KEY"),
    "secret_key": env.str("OVH_SECRET_KEY"),
    "bucket_name": env.str("BUCKET_NAME"),
    "region_name": env.str("BUCKET_REGION"),
    "endpoint_url": "https://s3.gra.io.cloud.ovh.net",
    "custom_domain": f"{env.str('BUCKET_NAME')}.s3.gra.io.cloud.ovh.net",
}
STORAGES = {
    "default": {
        "BACKEND": default_storage_backend,
        "OPTIONS": default_storage_options,
    },
    "public_media": {
        "BACKEND": default_storage_backend,
        "OPTIONS": {
            # add or override options defined in default
            **default_storage_options,
            "location": "media",
            "default_acl": "public-read",
            "file_overwrite": True,
            "querystring_auth": False,
        },
    },
    "private_media": {
        "BACKEND": default_storage_backend,
        "OPTIONS": {
            **default_storage_options,
            "location": "private_media",
        } 
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
    },
}

[ashwanthbalakrishnan]

Security Issue: AWS secrets keys revealed in migration files

Can anyone check if my configuration in this issues is correct
Or am I doing anything wrong ?

[Swannbm]

Yes, it is.

But migrations seems to have trouble generating correctly. You should edit file manually.