Image field shows as null after uploading

Using Django Mystery Errors
1

Hello!

I have my Django site wired up to use Google Cloud Storage to host image files and serve them through a Cloud CDN. Everything works as expected locally. I am able to upload a file in the Admin, see the file populate in GCS, then retrieve the url via a DRF API route. The problem only occurs when I deploy the service to Cloud Run. I am able to upload the image, see the upload populate in GCS, see the proper file name in the postgres DB, access the file from what should be the generated URL. There are no errors or anything, BUT when I try to look at the record in the admin it looks like there is no file, and the API always returns null.

I have checked and double checked all configurations and environment variables, IAM access, URLs, everything I can think of. Everything looks like it should be working properly. The uploads are going through and the main_image field is correctly populated in the Database.

Everything works perfectly with all of the same configuration values locally. I understand that it is entirely possible that the issue lies somewhere in Google Cloud and not Django but I am at a loss on where to go from here.

Thanks in advance for any help offered

2

I’d try to tackle the issue this way:

  1. check if the file really got properly stored and exists (either on the terminal if you have console access, otherwise write a small helper view in django returning file stats on the file’s path)
  2. your http server needs proper access to the uploaded file (at least read permissions), this is quite often an issue in containered distributions, where the app may run as a different user than the http server. To find out, checks the http servers logs and/or the file request’s header in the browser and adjust the user roles or permissions on the files.

Hope this helps.

3

Hey there! Thanks for the reply. I have confirmed that the file is correctly uploaded from the file field in the Django Admin from the cloud service, but returns to null when I view the record I uploaded it to. The DB has the proper file path. I printed out the MEDIA_URL on the server to make sure env vars were being loaded properly. The MEDIA_URL points to the correct domain and the filepath stored in the main_image column is correct. When putting the two together, I see the file. It seems like permissions are not an issue in this case as I confirmed the service account has the correct permissions to the Cloud Storage Bucket.

When I tried the following code, I got this error

 @action(detail=False, methods=['get'])
    def test(self, request, *args, **kwargs):
        services = Service.objects.exclude(main_image=None).all()
        result = []
        for s in services:
            main_image_str = str(s.main_image)
            main_image_name = getattr(s.main_image, 'name', None)
            main_image_exists = bool(main_image_name)
            if main_image_exists:
                try:
                    url = s.main_image.url
                except Exception as e:
                    url = f"ERROR: {str(e)}"
            else:
                url = 'NO FILE'
            result.append({
                "main_image": main_image_str,
                "main_image_name": main_image_name,
                "main_image_exists": main_image_exists,
                "main_image_url": url,
                "MEDIA_URL": settings.MEDIA_URL,
                "CDN_DOMAIN": settings.CDN_DOMAIN,
            })
        return Response(result, status=200)


The 'main_image' attribute has no file associated with it.

It seems to me there is some disconnect some where that is telling 2 different stories. The DB column is being populated on upload which causes main_image to not be null. But when trying to access main_image, it says there is no file associated to it. I am 98% certain that the permissions are correct considering the Django Admin is able to upload the images to the Storage Bucket, so I would imagine there would be no problem viewing them.

4

Does this error come from your exception clause, where you try to acess ImageField.url or from somewhere else? For all images or just a few?

To me it seems like an issue with the storage setup. Which storage class / settings do you use?

5

Finally found the issue… It seems my CDN is somehow setup to only allow retrieving files with signed URLs and that requires additional configuration to handle. Google Cloud Run will auth with the service account running the instance via token, but this isn’t enough to auth the signed URLs. This is why I was able to upload but not view from the Cloud Run instance.

ERROR: you need a private key to sign credentials.the credentials you are currently using <class 'google.auth.compute_engine.credentials.Credentials'> just contains a token. see Authentication — google-api-core documentation for more details.`

Thank you for the help debugging this, and sorry that it indeed ended up to be a “Doh” moment in my GCS configs. I was finally able to get the error to print out after disabling the DRF Camel Case plugin which was where the exception in my previous message was being thrown from.

All in all, I do think it would be nice if Django would have reported this error in retrieving the file rather than just showing it as null. Even if it were just to be logged as a warning.

6

Continuing the discussion from Image field shows as null after uploading:

Final solution was to update my STORAGES config to below

STORAGES = {
    "default": {
        "BACKEND": "storages.backends.gcloud.GoogleCloudStorage",
        "OPTIONS": {
            "project_id": os.environ.get("GS_PROJECT_ID"),
            "bucket_name": os.environ.get("GS_BUCKET_NAME"),
            "custom_endpoint": f"https://{CDN_DOMAIN}" if CDN_DOMAIN else None,
            "querystring_auth": False, # <--- ADDED THIS LINE
        }
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
    },
}

Although I had my CDN and Storage Bucket setup to allow public access without Signed URLs, Django Storages was requesting a signed version automatically via the querystring_auth option. Disabling that has fixed everything

7

Yes I second that. The question here is, at which level the error got swallowed (prolly in the storage class itself?)

Maybe your additional config setting is worth a note to the storage class maintainers and they can add that as a common pitfall to the docs?

8

It is really hard to pin down where in the trace things were going wrong/getting squelched. This was the exception I finally got to raise after some trial and error.

api/venv/Lib/site-packages/django/db/models/fields/files.py:42

    def _require_file(self):
        if not self:
            raise ValueError(
                "The '%s' attribute has no file associated with it." % self.field.name
            )

At first the file was just getting set to null, then, when running the test api route above, something about attempting to access the file fields url directly finally caused the above ValueError to be thrown.

Looking allllllll the way into googles cloud storage python lib, I see where the error is probably originating from, and there is a massive tree of things happening from there all the way up to Django then through to the views. A very long journey for a fringe exception to make.

It looks like the entire issue and solve for the problem is noted in the docs Google Cloud Storage - django-storages 1.14.6 documentation but finding that the signed urls were the issue is what really tripped me up. Knowing what to look for is half the battle.

I think I will chalk this one up to unlucky that I didn’t read the docs thoroughly and missed that the signed urls were enabled by default + the GCS/GCE pitfall. I would need to track down and understand the issue more to request any maintainer to further note it/update the error handling. Being that it ONLY happens when running from a GCS instance makes tracking it down a real nightmare.

Hopefully this thread can help a poor passerby in the future.

Cheers for the help again @jerch! Very enjoyable first interaction on the Django forums.

1 Like