After upgrading from django 3.0 to 3.1,
django.db.utils.InterfaceError: connection already closed issue started occuring randomly.
- gunicorn with 1 worker 6 threads
- postgresql version 13
- django version 3.1
- Using wsgi
Is it possible that the introduction of async support in django 3.1 is somehow introducing instability in DB connection? I don’t see any other feature in django 3.1 that might cause this issue.
I would also check and verify the versions of all other packages for being the most current and compatible set of packages for this installation.
Side note: I’ve not seen any issues like this with Django 3.2, but we don’t use gunicorn. We run Django in uwsgi behind nginx.
The issue disappears when changing Gunicorn to 1 worker 1 thread, or if the following signals are removed:
Upgraded Django to versions 3.2 and 4.0, but the issue persists.
Is it possible that we are facing a race condition due to the use of multithreading and the introduction of async on Django 3.1? We are not using async functionality on our code.
Actually, I’d be more likely to want to investigate what your “close_old_connections” handler does, and whether or not its functionality is affected by the upgrade. If that handler does anything with database connections, that’s likely the culprit.
(Or are you talking about the system-provided handler and not your own handler? If so, then that’s a different issue.)
I’m talking about the system-provided handler. I commented out, following the lead on this discussion: DB connection closed by uWSGI in the middle of handling a request?! - #2 by richardthegit, and got exactly the same outcome.
The only other comment I would make - and this is drawing from old experience which may no longer be true - is that we never run wsgi processes in a multi-thread worker. We only use the multi-worker environment where each worker runs a single thread. (This goes back to 2014 - Django 1.6 on Python 2, again acknowledging that a lot has changed since then - but this has been our common practice and we’ve never seen a reason to look to change that.)
What I’m seeing in the gunicorn docs is that they recommend either workers or threads to be set to " 2-4 x $(NUM_CORES) ", so neither setting really appears to provide a benefit over the other. And given that these processes are intended to be killed / restarted on a periodic basis, it does seem safer to me to run 1 thread per process.
It does seem reasonable to believe that the worker-restart process in gunicorn may not be fully thread-safe and that this issue is revealed by some change in Django 3.1.