Custom model superclass attributes that raise exceptions via DetailViews

Using Django Mystery Errors
1

I have a model superclass that adds a couple of attributes:

class MaintainedModel(Model):
    data = threading.local()
    data.default_coordinator = MaintainedModelCoordinator()
    data.coordinator_stack = []
    ...

Everything seems fine on the website until you go to any model’s DetailView, which raises an exception:

Traceback (most recent call last):
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/views/generic/detail.py", line 106, in get
    self.object = self.get_object()
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/views/generic/detail.py", line 52, in get_object
    obj = queryset.get()
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
    num = len(clone)
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/db/models/query.py", line 69, in __iter__
    obj = model_cls.from_db(db, init_list, row[model_fields_start:model_fields_end])
  File "/home/lparsons/mambaforge/envs/tracebase/lib/python3.8/site-packages/django/db/models/base.py", line 515, in from_db
    new = cls(*values)
  File "/home/lparsons/Documents/projects/tracebase/DataRepo/models/maintained_model.py", line 650, in __init__
    self._maintained_model_setup(**kwargs)
  File "/home/lparsons/Documents/projects/tracebase/DataRepo/models/maintained_model.py", line 660, in _maintained_model_setup
    coordinator = self.get_coordinator()
  File "/home/lparsons/Documents/projects/tracebase/DataRepo/models/maintained_model.py", line 1119, in get_coordinator
    return cls._get_current_coordinator()
  File "/home/lparsons/Documents/projects/tracebase/DataRepo/models/maintained_model.py", line 1123, in _get_current_coordinator
    if len(cls.data.coordinator_stack) > 0:

Exception Type: AttributeError at /DataRepo/samples/1/
Exception Value: '_thread._local' object has no attribute 'coordinator_stack'

I fixed the issue by adding the following to the top of _maintained_model_setup:

        if not hasattr(self.data, "default_coordinator"):
            self.data.__setattr__("default_coordinator", MaintainedModelCoordinator())
            self.data.__setattr__("coordinator_stack", [])

However, I’m not sure that’s the best solution, and I don’t really understand why the class attributes in this instance would not have been initialized. Can anyone explain why the data class attribute isn’t initialized when going through cls(*values) inside from_db and what may be the best way to avoid this exception?

2

The data object is local to a thread so its attributes set in one thread do not propagate to other threads.

When you set attributes on data in the class body, it is evaluated the first time the module is loaded, i.e. in the main thread.

So, those attributes exists in the main thread, but, as soon as a request is executed in another thread, those attributes are not defined.

I can’t say if your solution is the best way without knowing what is your intent with this thread local attribute

4

OK. Thanks for the explanation guys, and the suggestion. That makes a lot of sense. So what I’ve implemented is a context manager. The data attribute keeps the stack that the context manager uses to keep track of what the current context is. (It also keeps a default context instance. I thought that it would be a good idea to store those values in a thread-safe manner so that the stack is protected from multiple threads adding to it. Perhaps I need a way to initialize it in every thread?

It all works pretty well, actually, and is an enormous improvement from how I was doing it before (with globals).

Basically, if a model object is created inside a certain context, it behaves a little differently. I’ve disussed it with you before Ken, but the details aren’t terribly relevant here. I implemented the context manager based on the result of our discussion in another thread. If that’s not the way to do it, I’m not sure what’s better than that… I’d have to think about it.

5

I don’t get what you mean by context here. What does a context refer to ? A request ? A user session ? Something else ?

6

But to briefly explain the purpose of the context manager (and the super class), there are 3 different contexts. They all relate to the different possible behaviors of the superclass. The superclass adds the ability to automatically maintain the values of specific fields in records. If a field’s value is “maintained”, a derived model is not allowed to change/set it. When a record is saved, that field’s value is automatically generated. These changes can also propagate to other models whose maintained fields depend on the values of other fields.

When loading data, the generation of these values can go through pointless intermediate states, and depending on the derived class’s methods that generate the values, can be slow, so I implemented a context manager that has 3 different modes: immediate autoupdates (the slow version), deferred autoupdates (which skips the intermediate values, by buffering autoupdates until the last deferred context ends), and disabled autoupdates.

So, if the website is used to create/change a single record, the default context of “immediate” applies. We however have a validation interface, which runs their load in the disabled context (and using an atomic transaction context that rolls back everything in validation mode). And lastly, we have load scripts (not run on the website) that operate in the deferred autoupdate context.

7

Sorry, that’s not a very depthful explanation, but suffice it to say, it only applies to behavior of the superclass upon saving a record: whether it autoupdates maintained fields immediately, later (upon leaving the context), or never. All saved records behave in a manner consistent with the current active context.

8

So I have worked out a more thoughtful solution, thanks to the edification you guys provided…

My previous fix is a working hot fix and it’s only 3 lines of code. I think that a good follow-up update would be to remove direct access of the threading.local class attribute (data), e.g.: MaintainedModel.data.coordinator_stack and MaintainedModel.data.default_coordinator, from everywhere but 1 method, and in that method, address the thread initialization, if necessary. Initialization only ever needs to happen once, the first time it’s ever accessed. After that, it’s all managed automatically.

And I already have a method for coordinator default/stack initialization:

    @classmethod
    def _reset_coordinators(cls):
        cls.data.default_coordinator = MaintainedModelCoordinator()
        cls.data.coordinator_stack = []

but I currently only use it for testing.

And to allay your concerns, as I did in the original thread that inspired me to use a context manager, the default coordinator and the coordinator stack (for tracking context), are always reset when the last nested context is left, so they’re back to their initial state.

The other thing that I just realized while composing this reply is that the coordinator itself has 4 (essentially immutable once initialized) class attributes that I need to move to shared memory, so I’ll probably create a third class to hold that data.

Honestly, I’m pretty stoked to implement this fix. I just have some other work to get out of the way before I sink my teeth into it.