For the past few days, our Django website has been experiencing random PostgreSQL connection drops that would raise InterfaceError: connection already closed in random locations. The queries that cause the error occur in different views at random and we can’t replicate the issue reliably. It occurs very frequently, somewhere between 1-10% of requests that make DB queries.
All stack traces follow the same pattern. The only thing that differs is the location of our query that caused it. Here is an example stack trace.
InterfaceError: connection already closed
File "django/db/backends/base/base.py", line 301, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "django/db/backends/postgresql/base.py", line 269, in create_cursor
cursor = self.connection.cursor()
InterfaceError: connection already closed
File "django/core/handlers/exception.py", line 56, in inner
response = get_response(request)
File "django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "contextlib.py", line 79, in inner
return func(*args, **kwds)
File "django/views/generic/base.py", line 103, in view
return self.dispatch(request, *args, **kwargs)
File "django/contrib/auth/mixins.py", line 73, in dispatch
return super().dispatch(request, *args, **kwargs)
File "django/views/generic/base.py", line 142, in dispatch
return handler(request, *args, **kwargs)
File "tolink/views/app.py", line 273, in get
self.set_profile(pk)
File "tolink/views/app.py", line 267, in set_profile
profile = Profile.objects.filter(pk=pk, user=self.request.user).first()
File "django/db/models/query.py", line 1047, in first
for obj in (self if self.ordered else self.order_by("pk"))[:1]:
File "django/db/models/query.py", line 394, in __iter__
self._fetch_all()
File "django/db/models/query.py", line 1867, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "django/db/models/query.py", line 87, in __iter__
results = compiler.execute_sql(
File "django/db/models/sql/compiler.py", line 1396, in execute_sql
cursor = self.connection.cursor()
File "django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "django/db/backends/base/base.py", line 323, in cursor
return self._cursor()
File "django/db/backends/base/base.py", line 300, in _cursor
with self.wrap_database_errors:
File "django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "django/db/backends/base/base.py", line 301, in _cursor
return self._prepare_cursor(self.create_cursor(name))
File "django/utils/asyncio.py", line 26, in inner
return func(*args, **kwargs)
File "django/db/backends/postgresql/base.py", line 269, in create_cursor
cursor = self.connection.cursor()
Details of our setup:
- Django 4.1.7
- gunicorn 20.1.0
- psycopg2 2.9.5
- PostgreSQL 14.6
- Heroku basic dyno (hosting the django server)
- Basic Heroku Postgres (hosting PostgreSQL)
The portion of settings.py that’s relevant to the DB looks like this:
CONN_HEALTH_CHECKS = True
CONN_MAX_AGE = None
DATABASES = {
"default": dj_database_url.config(
conn_max_age=CONN_MAX_AGE,
conn_health_checks=True,
ssl_require=env("DATABASE_SSL_REQUIRED"),
)
}
DATABASES["default"]["ATOMIC_REQUESTS"] = True
I found some old stack overflow and Django forum questions that seemed to experience similar problems, however, their solutions were specific to their stack (nginx + uwsgi). Unlike in other questions, PostgreSQL connection limit seems to not be the problem in this case (2 worker threads only with a limit of 20 connections). These problems occur in production, not during testing or local development.
I tried updating the dependencies, restarting the database/server, and played around with database settings.
Even if you haven’t experienced something similar before, I’d appreciate if you could give guidance on how I can investigate this issue.