Say I upload images related to an object to a /media folder in a bucket, the access for which is not supposed to be public. The upload is managed via S3Boto3Storage and does upload the images fine. Locally (e.g. not uploaded to S3), I can display the images in my templates:
And that displays my image (since it’s just stored locally via the FileSystemStorage).
However, if I try to run the same with the actual S3 backend, then the image uploads fine to the bucket, but the browser displays an empty content for the <img> tag. On the console I get 403 Forbidden. And the headers for the request show none of the credential/token exchange to tell S3 I actually have the rights to access this. Makes sense.
But… is there a middleware to manage this aspect? Given that all the credentials issues are managed by django for the upload (given proper settings.py / AWS configs), I’m guessing there has to be a way for django to also insert the proper credentials in the headers so that it would display the images as expected?
Not so much middleware (from the Django perspective), but perhaps use nginx as the proxy.
You could configure nginx with the authentication credentials for the s3 bucket, then configure your static urls to refer to a configuration section in nginx that pulls files from the bucket.
The end users never see an s3 url, the s3 bucket remains protected, and if you need to restrict access to any of those files to authenticated users, you have the rest of the nginx facilities to manage access to those urls.
Ah yes that could work, I’m already running Nginx as a reverse proxy, so it would be a matter of configuring the auth I guess. And Nginx does have a blog writeup about something along those lines.
That being said, I managed to setup something that doesn’t require nginx. In fact my initial setup was pretty close, but I was missing a somewhat obscure detail that prevented the url saved in the db as the location of the ImageField from containing the necessary keys in that url.
I still have an issue though - which is that unless I configure those to not expire, I think the parameters from those url eventually will, so they won’t be valid eternally. So I might have to use Nginx anyways as you proposed.
I’ll add more details in the question about what I did when I have time.
Take a look at S3 Signed URLs which you can generate just in time to have this work as expected. If you’re using django-storages this can be configured for media files relatively easily. This article has a nice example and runs through all the different ways of using S3 with static and media files: Storing Django Static and Media Files on Amazon S3 | TestDriven.io
Yeah I saw (aws’s docs) about presigned urls as well. I’ll admit to having been a little lazy and not given it a very complete/serious consideration. Largely because:
That does mean that each and every view that wants to handle those documents will have to implement this. I would wrap that in a Mixin, but still.
From what I understand, I need to generate the presigned url on the fly. So in a way, that does mean that either I’m tried down to that API version and/or I commit to maintenance if they decide to change how their api work. After all, previous versions used to allow tokens without expiration for this. Now they don’t.
I kinda wished they didn’t decide for me whether or not tokens without expirations are safe enough of not.
But perhaps I’ll revisit that approach I guess. Though if nginx manages the auth for this, then I don’t really have to care about it anymore on the django side.
Oh actually, I think I checked that too quickly. If I understand correctly, PrivateMediaStorage() would manage for me the business of creating the presigned url. And so if that is true, then my main gripes about using that approach go away - it’s actually Boto3 who takes care of keeping things working with different version, and I don’t need a Mixin or something to add to each view using privates object.
I do wish the docs fro S3Boto3Storage was a bit more exhaustive than this. They do mention CloudFront signed url, but it really wasn’t all that obvious to me at least that this allowed something like the link you put up. Nor does it seem to me it really gives all the info you need to figure it out yourself…