Where is the image file?

Using Django Using the ORM
1

Hi all,

I have this model:

class Product(VASTObject_NameUserGroupUnique):
    image                = models.ImageField(upload_to='product_images/', default=None, null=True, blank=True)
    image_resource_id    = models.IntegerField(default=None, null=True, blank=True)
    image_uriref         = models.URLField(max_length=512, null=True, blank=True)

    def save(self, *args, **kwargs):
        logger.info(f"Product: save():", *args, **kwargs)
        if self.image:
            path = self.image.path
            url  = self.image.url
            logger.info(f"Product: save(): Image url:  {url}  (exists: {os.path.exists(path)})")
            logger.info(f"Product: save(): Image path: {path} (exists: {os.path.exists(path)})")
            if not os.path.exists(path):
                # Image not found at this path. Look into product_images...
                head_tail = os.path.split(path)
                path = os.path.join(head_tail[0], 'product_images', head_tail[1])
                logger.info(f"Product: save(): Trying new Image path: {path} (exists: {os.path.exists(path)})")
                if os.path.exists(path):
                    head_tail = os.path.split(url)
                    url = os.path.join(head_tail[0], 'product_images', head_tail[1])
                    logger.info(f"Product: save(): Trying new Image url:  {url}  (exists: {os.path.exists(path)})")
        super().save(*args, **kwargs)

When I create a new product, it seems to work. But when changing the image on an existing object, I get this output:

Product: save():
Product: save(): Image url:  /media/20230520_174836.jpg  (exists: False)
Product: save(): Image path: /backend/media/20230520_174836.jpg (exists: False)
Product: save(): Trying new Image path: /backend/media/product_images/20230520_174836.jpg (exists: False)

The fields in image seem correct, but the actual file is not in the media folder.
Where is my image file?
(I need o upload it into a digital assets management system).

2

There are a couple of points to keep in mind when working with uploaded files:

  • Files are stored in a physical path based on MEDIA_ROOT, but accessed through the logical path based on MEDIA_URL. The two do not need to have any relationship to each other.

  • Files may have their name changed when being saved. Django will not overwrite one uploaded file with another. It’s up to you to keep track of either the pk of the entity referencing that file or saving the original name used for the upload.

Also, the file is committed to the storage during the pre_save step of the model being saved. Since you’re trying to do this in the models save method, you’re (possibly) looking for the file before it has been written to MEDIA_ROOT.

3

And how can I add code to the pre_save step? It seems it is a signal?

4

If I save the image myself, upload the image into the DAM, and then delete the image, before calling super().save(), seems to work.

5

No, from what I can see in the code (I’m not sure of this), the “pre_save” step in the save process is just the name given to functions that are called for each field while doing the save. It’s not tied to the signals processes.

You can call your logging code after the call to super().save(...) - the data should be there.

6

Well, because I need the image before save(), what I did is to save a temporary image, and remove it:

    def create_image_resource(self):
        if self.image:
            path = self.image.path
            url  = self.image.url
            logger.info(f"{self.__class__.__name__}: create_image_resource(): Image url:  {url}  (exists: {os.path.exists(path)})")
            logger.info(f"{self.__class__.__name__}: create_image_resource(): Image path: {path} (exists: {os.path.exists(path)})")
            delete_tmp_image = False
            if not os.path.exists(path):
                ## Try to save the image in a temporary file...
                path = self.image.path
                logger.info(f"{self.__class__.__name__}: create_image_resource(): Saving Temp Image: {path}")
                # Open the image using PIL
                img = Image.open(self.image)
                img.save(path)
                delete_tmp_image = True
            if os.path.exists(path):
                dam = DAMStoreVAST()
                logger.info(f"{self.__class__.__name__}: create_image_resource(): Creating Image resource...")
                self.image_resource_id = dam.create_resource(self.image.url, {
                    'description': f'{type(self).__name__}: {self.name}',
                })
                logger.info(f"{self.__class__.__name__}: create_image_resource(): Image Resource: {self.image_resource_id}")
                json_data = dam.get_resource(self.image_resource_id)
                self.image_uriref = dam.get_size(json_data)['url']
                logger.info(f"{self.__class__.__name__}: create_image_resource(): Image url: {self.image_uriref}")
                del dam
                if delete_tmp_image:
                    logger.info(f"{self.__class__.__name__}: create_image_resource(): Deleting Temp Image: {path}")
                    os.remove(path)
            else:
                self.image_resource_id = None
                self.image_uriref      = None

Using this temporary image I can push it to an external DAM, get the URL in the DAM, and save it in the model.

7

Ok, it seems images were easy, saving temporarily the files is more complicated.

document = models.FileField(upload_to='stimulus_documents/', default=None, null=True, blank=True)

    def create_document_resource(self):
        if self.document:
            path = self.document.path
            url  = self.document.url
            logger.info(f"{self.__class__.__name__}: create_document_resource(): Document url:  {url}  (exists: {os.path.exists(path)})")
            logger.info(f"{self.__class__.__name__}: create_document_resource(): Document path: {path} (exists: {os.path.exists(path)})")
            delete_tmp_document = False
            if not os.path.exists(path):
                ## Try to save the document in a temporary file...
                path = self.document.path
                logger.info(f"{self.__class__.__name__}: create_document_resource(): Saving Temp Document: {path}")
                # Save the file on disk...
                with open(path, 'wb') as fd:
                    with self.document.open('rb') as file:
                        fd.write(file.read())
                delete_tmp_document = True

The file is saved, I can upload it to the digital assets management, but django cannot save any more the instance to the database:

  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1695, in pre_save_val
    return field.pre_save(obj, add=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/files.py", line 317, in pre_save
    file.save(file.name, file.file, save=False)
  File "/usr/local/lib/python3.11/site-packages/django/db/models/fields/files.py", line 93, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/files/storage/base.py", line 38, in save
    name = self._save(name, content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/files/storage/filesystem.py", line 101, in _save
    file_move_safe(content.temporary_file_path(), full_path)
  File "/usr/local/lib/python3.11/site-packages/django/core/files/move.py", line 61, in file_move_safe
    with open(old_file_name, "rb") as old_file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp9kkwzab5.upload.pdf'

So, the question is, how can I save a temporary copy of a FileField, before save() is called?

8

It is not entirely clear to me as to why you need the file pre-save. From what I read I deduce that you want to save the file in a DAM instead of the media folder of your Django app.

Options:

  • don’t use an image field in the model but a url field for example and handle the upload of the image to the DAM yourself
  • perhaps the DAM can be used as the Django media location. For example you can have an AWS S3 bucket as your media files location
  • you can also extend the middleware layer in Django to handle the upload to your DAM perhaps by creating your own custom field type
9

I want to save the image in an external DAM (ResourceSpace), and update the instance with the DAM resource id & URL in the DAM. ResourceSpace allows only import of resources from an online url, so I need the image to be accessible in django through a url.

I will keep images in both django & DAM.

the problem is that the file is uploaded by the user in a form. The file is nowhere to be found though, until I save the model instance. But then I need to modify instance, and save again?

Right now, the best approach I have found that seems to work, is to save the FileField (from inside the model) before saving the instance, like this:

self.document.save(self.document.name, self.document.file, save=False)
--- save in DAM & update other model fields ---
super().save
1 Like
10

Why not upload the file to the DAM after save. You can still make it a single transaction and rollback the save should the upload fail and update the record with the DAM uri.

11

Because I need to save info from the DAM to the database (its id in DAM and its url).
So as to display it, and delete the DAM entry if the django db object gets deleted.

12

You could explore option 3 from my previous reply.

13

Its a very complex option. It should have been much easier in django in my opinion.
Right now, the solution I am using (saving the field with save=False) its much easier.
Its not documented, but…

14

It is documented Creating forms from models | Django documentation | Django

You can look for the commit=False

This save() method accepts an optional commit keyword argument, which accepts either True or False . If you call save() with commit=False , then it will return an object that hasn’t yet been saved to the database. In this case, it’s up to you to call save() on the resulting model instance. This is useful if you want to do custom processing on the object before saving it, or if you want to use one of the specializedmodel saving options. commit is True by default.

However I don’t know how it effects the file saving but I suspect the file will be stored in the media folder despite that there is no commit. Probably a good idea to use atomic to treat all your operations as 1 transaction so that you don’t get any inconsistencies like having files in the media folder that do not have a corresponding record.