It’s perfect that this came in part way through my talk at Djangocon!
A lot of great points to digest, so here goes:
The multiple backend support is one which I’ve gone back and forward on multiple times.
The idea behind Task.backend is to limit which backend a task can run on. I see it akin to specifying the database for a given model. A task must therefore only be run on the specified backend, which is what’s validated in validate_task. I don’t think we can remove validate_task, as it does more than just validate the applicability of the task.
Multiple backends isn’t much additional work to maintain, but it may not be worth these odd semantics. Highly open to opinions here.
“Public” here is a little ambiguous. I completely agree that it should be removed from the usage documentation for enqueueing tasks. However, it will still need documenting somewhere for those implementing a backend themselves (where enqueue is the main entrypoint).
I’m not sure if Django has a distinction between those things, or how best we’d handle it. But I think we’re on the same page.
In hindsight (what a great thing), I agree. In principle, it makes sense as something to offload, but I agree the return value makes it something of a dealbreaker. The current implementation simply reports that all messages sent successfully, which is obviously not correct.
Opened an issue.
I have this implemented. The implementation doesn’t store anything in memory, instead using the run_after as part of the scheduling, sorting first by priority, then by run_after.
I don’t think this is correct, as tasks with a run_after may never end up getting scheduled. However, that’s likely correct. A task with a higher priority should be run before one with a lower priority but with a run_after date. It’s specifically called run_after rather than run_at because of the unpredictability. That could absolutely be a foot-gun though.
If there was a cron system already, I’d agree it could be worked around, however it’s still a very useful piece of functionality, so long as we can document the queueing semantics well enough.
Fantastic idea! Opened an issue.
Connection management is a tricky one. The pattern was copied from the cache backends, which handle connections with this pattern. Lazy initializing the client adds more complexity, but it does make checking and more static analysis much simpler.
Given the built in versions won’t handle any connection management (and won’t with the defined stretch features), I agree it makes sense to remove, let backend developers implement their own pattern, since it’s likely to vary heavily per backend.
I’ve opened this as a discussion to collect input,
I’m not sure whether the forum is the best place for these discussions (I assume probably not). If not, please feel free to open the relevant discussions/issues on the repo, or grab me IRL.
(There’d be more links, but Discourse said no)
!