Common jobs/queue API?

Thanks Django team for building such an amazingly useful and durable framework!

When I started using Django, it was a little overwhelming to have to sort through all the different third-party offerings for task/job queues. And now that I have several large projects depending on specific libraries, I sometimes worry that I picked the wrong one, in terms of long-term support and bug fixes.

I think would be nice to have a high-level jobs API built into Django, similar to Active Job for Rails. As those docs say:

… without having to worry about API differences between various job runners such as Delayed Job and Resque. Picking your queuing backend becomes more of an operational concern, then. And you’ll be able to switch between them without having to rewrite your jobs.

Is this something that could be in Django’s future?

This has definitely been a recurring idea through the years, but I don’t think it’s ever been particularly hammered on - job queuing systems tend to have such different abstractions (at the very least, the choice between at-least-once and at-most-once) that it’d be tough to unify them all.

I also think Celery’s dominance for many years plays into this as well - it just plugged into Django, obliviating the need for too much else.

If you’re interested in the idea, I’d say this - what do you think the common denominator API would be here? API design is probably the most important piece of work here; it has to be powerful enough to allow different systems to actually be useful, but common enough that, as you say, it can be switched between easily.

I think the API would probably be useful even if it only had the following:

  • define multiple queues
  • schedule a job for X time in a particular queue
  • run job in a particular queue
  • management command to start/stop workers
  • a way to test jobs by running them immediately and wait for results
  • view jobs in admin, where they can be deleted or retried

For use cases outside the API’s scope, the developer could interact directly with the underlying library, or the API could expand over time as it becomes clearer how to navigate the underlying differences. I can see how trying to support too wide a range of features would be problematic. However, I think there are many use cases that only need a narrow range of features, based on the proliferation of job libraries that advertise themselves as “easier to use than Celery”.