Cache improvements (reuse connections + memcached pooling)

Django Internals Mentorship
1

Hi everyone.

Although i’m no Django expert, i would like to contribute a piece of code to Django, and i was thinking on cache.

I’ve seen two tickets around a problem with cache connections not being reused.

What i could understand from the code, django uses the CacheHandler class with an asgiref.Local to store connections to the cache implementations, however, each request with a new thread/task will end up creating a new connection, because the Local won’t find the previous ones.

This seems problematic when there are caches that already provide some protection mechanisms, like locmem Lock, redis connection pools, etc.
Another problem, regarding memcached, it seems pylibmc has been unmantained for 2 years, (found it after it failed on a local project). pymemcache on the other hand, works, but if you configure connection pooling (passed via OPTIONS), there is a signal in CacheHandler to close the client after each request, so you end up losing the benefit.

I would like to know how would you tackle this problems.

  1. Maybe we could modify CacheHandler to use singletons instead of asgiref.Local and delegate the protection to the cache implementations (maybe an attribute could be used to define which caches are safe, and which not).

  2. is it worth mantaining pylibmc? (there are bugs not being solved)

  3. work around connection pooling support in pymemcache with: HashClient(..., use_pooling=True) to do not close the client on request finish. If the cache OPTIONS contain pool_idle_timeout > 0, do not close the client.

Had to remove pylibmc github and hashclient docs, because of 2 link limitation. but they exist.
Ty!