django-tasks interfaces with other background worker systems

Django Internals
1

I’ve watched the excellent talk from DjangoCon Europe about the new django-tasks system, which will make its way into Django core at some point relatively soon. I was not entirely clear on the relationship between the standardized worker API and the worker execution systems like Celery or RQ. Initially I believe the goal of the API is just to have a simple database-backed worker queue system, but then is the goal to later provide interfaces or backends which can pass a job to Celery or similar?

Thanks!

2

Let’s get Jake to input here: @theorangeone .

Both are goals. I think it would be good to clarify which is the primary one. I would like the built-in database-backed worker to be the priority since that’s something we have more control over and can provide for most users.

1 Like
3

The goals are 2-fold:

  1. Build an API contract for other backends to be plugged in
  2. Provide a few useful backends in Django core

“useful” here is strictly standalone. Celery or RQ backends won’t end up in core, nor will any other 3rd-party adapters. The API contract is a fair chunk of the power, and has to be in core itself to be useful. But the value for newcomers and smaller projects is to a production-grade backend built-in. Of which the ORM backend will be the first, and then I hope to write a Redis one in future.

Both are very much goals, and I don’t think it’s useful to prioritise them. The API contract must push forwards so that the DB backend can. If the DB backend needs to work around the API contract, then the contract has failed.

1 Like