dumpdata error in FieldCacheMixin ?

Hi all - I’ve put a couple of days into completing [model_instance].natural_key() function, natural_key.dependencies list and a [model_class].objects.get_by_natural_key() function. When I invoke with:

python manage.py dumpdata auth accounts bwm_revised criteria_prioritisation dcap info projects prioritisation_info --natural-primary --natural-foreign -
-indent 4 -v 2 --traceback -o a_to_d.json

I get an error trying to when related_descriptors.py tries to use fields/mixins.py which exceeds the recursion depth. The task half-finishes, then throws this error:

[… ]Traceback (most recent call last):
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\db\models\fields\related_descriptors.py”, line 174, in get
rel_obj = self.field.get_cached_value(instance)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\db\models\fields\mixins.py”, line 16, in get_cached_value
return instance._state.fields_cache[cache_name]
KeyError: ‘from_content’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “C:\Users\atcra\Coding\KEN_development\manage.py”, line 22, in
execute_from_command_line(sys.argv)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\core\management_init_.py”, line 419, in execute_from_command_line
utility.execute()
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\core\management_init_.py”, line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\core\management\base.py”, line 354, in run_from_argv
self.execute(*args, **cmd_options)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\core\management\base.py”, line 398, in execute
output = self.handle(*args, **options)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\core\management\commands\dumpdata.py”, line 232, in handle
serializers.serialize(
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\core\serializers_init_.py”, line 129, in serialize
s.serialize(queryset, **options)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\core\serializers\base.py”, line 110, in serialize
self.handle_fk_field(obj, field)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\core\serializers\python.py”, line 53, in handle_fk_field
related = getattr(obj, field.name)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\db\models\fields\related_descriptors.py”, line 188, in get
rel_obj = self.get_object(instance)
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\db\models\fields\related_descriptors.py”, line 155, in get_object
return qs.get(self.field.get_reverse_related_filter(instance))
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\db\models\query.py”, line 435, in get
raise self.model.DoesNotExist(
info.models.DoesNotExist: Content matching query does not exist.
Exception ignored in: <generator object cursor_iter at 0x0000026EB26A2EB0>
Traceback (most recent call last):
File “C:\Users\atcra\Coding\KEN_development\devenv\lib\site-packages\django\db\models\sql\compiler.py”, line 1649, in cursor_iter
cursor.close()
sqlite3.ProgrammingError: Cannot operate on a closed database.

Django 3.2.4 django.db.models.fields.mixins defines:

class FieldCacheMixin:
“”“Provide an API for working with the model’s fields value cache.”“”

def get_cache_name(self):
    raise NotImplementedError

def get_cached_value(self, instance, default=NOT_PROVIDED):
    # print("get_cached_value, type:", type(instance), 'value:', instance, "\n")
    cache_name = self.get_cache_name()
    try:
        return instance._state.fields_cache[cache_name]
    except KeyError:
        if default is NOT_PROVIDED:
            raise
        return default

This is the ContentTrace model that has the ‘from_content’ field that couldn’t be found:

class ContentTrace(BaseModelWithTags):
from_content = models.ForeignKey(Content, on_delete=models.CASCADE, related_name=“traces_from”)
to_content = models.ForeignKey(Content, on_delete=models.CASCADE, related_name=“traces_to”)
objects = discrete_manager_factory(“N/A”)
natural_key_fields = (‘from_content__content_type__app_label’,
‘from_content__content_type__model’,
‘from_content__object_id’,
‘to_content__app_label’,
‘to_content__model’,
‘to_content__object_id’)
natural_key_dependencies = [‘defenceinfo.content’,]

def __str__(self):
    return 'from:' + str(self.from_content) + ", to:" + str(self.to_content)

There are no instances of ContentTrace or Content objects at the moment; I don’t believe that could be the problem, but I’m out of ideas.

Hi atcrank,

Does dumpdata work when you don’t limit the applications? If it does, it sounds like you need to include the app that has ContentTrace and Content.

Hi Tim, thanks for responding. Unfortunately (for me) that field is in a model that is in the ‘info’ app.

My current suspects:

  1. data in dev database is corrupt / inconsistent with the models, e.g. I’ve accepted a migrations option of ‘allow nulls to remain and deal with them later’ and then not dealt with them later.
  2. with different apps selected, different fields trigger this error, but it has always been a ContentTypes-based field. Possibly we have some wrong usages of GenericRelation or GenericForeignKey. (But it all works, aside from this issue).
  3. a bug or effect of ‘not implemented’ in the Mixin.
  4. different apps reusing model and field names (this would not explain all the errors though)

I think I need to level up my debugging.

It seems to be related to django-simple-history Historical objects with foreign keys that are now deleted objects. They exist only in the Historical version of the related table, but the serializer doesn’t know to call the alternate model manager to find them. I learned a lot finding this out, but not much about what to do about it (except maybe put all the Historical records in custom apps so I can easily exclude them from dumping and loading.)

I’m attempting to move the Historical models from django-simple-history into their own ‘history’ database, which seems to be working tolerably well. But that was not the answer, as the historical models already in the ‘default’ database are still being exported and causing the process to crash.

If they record a reference (e.g. ForeignKey) to model that has been deleted in the live versions, the serializer doesn’t realise it could find the historical version of that object. My preferred solution would be for deletion of models to correspond to changing that reference; or for the history to be compiled so that all of those relations are paralleled by relations to Historical models. (LMK if I have missed this option somehow; I know I have the option to cascade deletes, but that is not going to satsify the reqt.)

Another possibility would have been setting a custom app_label, but that created the error:

history_tracker.HistoricalVote.question was declared with a lazy reference to ‘history_tracker.question’, but app ‘history_tracker’ doesn’t provide model ‘question’.

I don’t think I can shoehorn all the models into a single app.

So it looks like I will be limiping along writing a dumpdata command with a hundred separate --exclude entries.

Could you write a wrapper around dumpdata that would generate the history model exclusions for you based on the existence of a new parameter?

python manage.py dumpdata ...apps --exclude-history ...params -o a_to_d.json

That sounds like a great approach, thanks Tim. Django-extensions graphmodels has a --X / --I parameter which takes wildcards, that does pretty much what I want (but for .dot graphs.)