Setting CONN_MAX_AGE made my Django task workers 3 to 4x faster

Using Django
1

I’ve been benchmarking database-backed task workers and found something worth sharing whichever backend you use.

With Django’s default CONN_MAX_AGE = 0, a worker closes its database connection after every task, so every task opens a new PostgreSQL connection. In my runs that cost about 5 ms each, more than the claim-and-run path takes on a warm connection.

Four workers, 20,000 no-op tasks, PostgreSQL 16 in Docker on the same host, django-ox 1.9.0 and django-tasks-db 0.13.0, medians of five runs:

  • CONN_MAX_AGE = 0: 430 tasks a second on django-ox, 373 on django-tasks-db
  • CONN_MAX_AGE = 600: 1,771 and 1,211
  • psycopg pool via OPTIONS["pool"]: 1,718 and 1,216

Holding the connection is about a 4x change for django-ox and 3x for django-tasks-db. Django 6.0 rejects a pool combined with a nonzero CONN_MAX_AGE, so pick one. If your worker runs the default, that’s the first thing I’d change before adding more processes.

Both backends got identical connection settings. With a pool or persistent connections, django-ox did 1.4 to 1.7 times the tasks per second of django-tasks-db. These tasks do nothing, so real tasks that spend their time in your code will narrow the gap. Settings and every raw number: Benchmarks - django-ox

django-ox is 100% open source (BSD-3-Clause) with no paid tier. Outside contributors have merged fixes, and pull requests are welcome: django-ox/CONTRIBUTING.md at main · oxpull/django-ox · GitHub

If you get different numbers on your own hardware, I’d like to see them.