I have proposed an initial PR that I’m happy with, but the question of whether functions wrapped with functools.cache (or lru_cache) would be accepted as valid callables (ie whether the migrations serializer would be able to serialize them) still needs addressing. The answer so far is “no” because the mentioned functools decorators return an instance of _lru_cache_wrapper (see this post with details in the Python discourse), so I was wondering if we would be open to do either of:
Provide a specific serializer for the _lru_cache_wrapper type, or
Provide a serializer for anything that has a __wrapped__ attribute
I think supporting this use case (functions wrapped in caching helpers to be passed as callables) is important, since one of the reasons I can think of for wanting callable choice is to defer a potentially expensive calculation (or I/O-bound operation) that would benefit from caching as well.
I’m in favour of doing something here. My main question with option 2 is can we serialize the decorator stack as well as the innermost callable? If not, option 1 sounds simpler and perhaps good enough.
I’m assuming that in the migrations we want to call the fully decorated callable, not just the innermost callable.
My understanding of the migrations machinery is incipient, therefore I was assuming that any evaluation of the callable at migration time would be in a separated process, thus even if the “caching version” of the callable is used, the cache access would “always” be a MISS since there is no run that has pre-populated.
I think you’re right that in the case of a caching decorator it’s fine to omit it, so that works for option 1.
But for option 2, we don’t know what the decorators do - it’s possible that they’re needed for correctness.
So I think there isn’t really anything to worry about with respect to option 2 as long as decorators are well defined using functools.wraps() to forward __module__ and __qualname__, etc. In the migrations serializer we’re only ending up with the dotted path to the function and the module to import, not a serialized copy of the actual function code. For a decorated function you would still import it normally by the name you’d expect:
Ah, I realised that this wasn’t quite fair because I’ve skipped the detection bit and manually assumed these would be identified as functions. So better would be the following instead which shows that the general decorator case using functools.wraps still works fine:
But, yes, it seems that the @functools.(lru_)cache case will need some help:
In [2]: from functools import cache
...: from django.db.migrations.serializer import serializer_factory
...:
...: @cache
...: def function():
...: pass
...:
...: serializer_factory(function).serialize()
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[2], line 8
4 @cache
5 def function():
6 pass
----> 8 serializer_factory(function).serialize()
File ~/Sources/django/django/db/migrations/serializer.py:396, in serializer_factory(value)
394 if isinstance(value, type_):
395 return serializer_cls(value)
--> 396 raise ValueError(
397 "Cannot serialize: %r\nThere are some values Django cannot serialize into "
398 "migration files.\nFor more, see https://docs.djangoproject.com/en/%s/"
399 "topics/migrations/#migration-serializing" % (value, get_docs_version())
400 )
ValueError: Cannot serialize: <functools._lru_cache_wrapper object at 0x7fae891a2350>
There are some values Django cannot serialize into migration files.
For more, see https://docs.djangoproject.com/en/dev/topics/migrations/#migration-serializing
It seems that we can quite happily extend this:
In [3]: from functools import _lru_cache_wrapper
...: from django.db.migrations.serializer import FunctionTypeSerializer, Serializer
...: Serializer.register(_lru_cache_wrapper, FunctionTypeSerializer)
And it works!
In [4]: from functools import cache
...: from django.db.migrations.serializer import serializer_factory
...:
...: @cache
...: def function():
...: pass
...:
...: serializer_factory(function).serialize()
Out[4]: ('__main__.function', {'import __main__'})
