Task Cancellation

Django Internals
1

Currently, there is no supported way to stop a task from running once it has been enqueued. Adding first-class support for this would allow application developers to build UIs for canceling pending tasks (which is my use case that brings me here).

Various task systems support preventing a task from running or stopping one that is running. Properly representing that state, rather than just using FAILED, allows application developers to differentiate a task that was stopped from one that crashed.

I propose that Django add to TaskResultStatus: CANCELLED and ABORTED. Both qualify as TaskResult.is_finished. Backends set the status, update TaskResult.finished_at, and fire the task_finished signal when they know the task has stopped. TaskResult.return_value would raise ValueError("Task cancelled") or ValueError("Task aborted"), respectively. TaskResult.errors would be empty, unless an exception occurred for some other reason, e.g. as part of aborting the running task.

Backwards-compatibility:

  • Existing code that treats status == FAILED as the only possible failure will miss the new statuses.
  • Existing code that assumes is_finished and status != SUCCESSFUL means a crash is fine in practice. It overstates for the new statuses, but overstating is less problematic than understating. How much it overstates is situation specific: A task aborted midway through may be no different than a crash or it may have reverted things in an orderly fashion. A task that was cancelled didn’t crash, but also didn’t do what was asked.
  • Existing code that does an exhaustive match (e.g. with assert_never or raise NotImplementedError) will need to support the new statuses. But that is the expected and desired result of using that pattern.

Allowed state transitions:

  • READY → CANCELLED
  • RUNNING → ABORTED
  • READY → ABORTED (only when the backend cannot tell whether the task had started)

If the backend does not support the more specific states, they could continue to map to FAILED.

Spelling: I am proposing CANCELLED rather than CANCELED because that matches asyncio.CancelledError in Python and Procrastinate’s CANCELLED.

I think two separate states are warranted for several reasons:

  • This maps better to existing task systems.
  • ABORTED indicates that the task was stopped mid-execution. It could have made incomplete changes.
  • CANCELLED plus started_at is not None is not the same as ABORTED if a backend supports retries. If a task runs, fails, is set up to retry (without resetting started_at), and then cancelled, it would be in the state CANCELLED plus started_at is not None, but it was not aborted mid-execution.

I think the state mappings would look like this:

Celery:

  • Celery seemingly allows .revoke() to either prevent execution of a pending task or send a signal to a running task. Both will end up as REVOKED.
    • A CeleryBackend may be able to 1) call revoke() and set TaskResultStatus.CANCELLED, and 2) call revoke(terminate=True) and set TaskResultStatus.ABORTED.
    • If it can’t do that and has to naively convert Celery REVOKED, that should map to TaskResultStatus.ABORTED because it can’t rule out that the task had started.
    • If it implements Celery’s cooperative abort, Celery’s ABORTED status only indicates that the request was made; this maps to TaskResultStatus.RUNNING until the task has actually stopped. Only then can it become TaskResultStatus.ABORTED.

Chancy:

  • No separate states, so both scenarios remain using FAILED.

django-tasks-db:

  • This is what I use, and what brought me here. Adding a CANCELLED state is relatively easy.
  • It would likely not support a separate abort.

Procrastinate:

  • CANCELLED → CANCELLED
  • ABORTED → ABORTED

RQ:

  • canceled → CANCELLED
  • stopped → ABORTED

I propose the following API methods:

  • BaseTaskBackend.abort(result_id): Attempts to stop a task, which may or may not be running. If the task does not exist, TaskResultDoesNotExist will be raised by backends that supports_get_result. If the task is known to already be ABORTED or CANCELLED, this method will return successfully. If the backend knows it cannot stop this task, TaskNotStoppable is raised. (This includes tasks that have already finished as SUCCESSFUL or FAILED.) When this method returns, the stop has been requested, but may not be complete. If the backend is eventually able to stop the task, its status will become CANCELLED if the backend knows the task never started, otherwise ABORTED.
    • Backends that abort tasks synchronously should document that.
    • The default implementation of BaseTaskBackend.abort() would be to call self.cancel().
  • BaseTaskBackend.cancel(result_id): Attempts to stop a task, only if it has not started running. If the task does not exist, TaskResultDoesNotExist will be raised by backends that supports_get_result. If the task is known to already be CANCELLED, this method will return successfully. If the backend knows it cannot stop this task, TaskNotStoppable is raised. (This includes tasks that have already finished as SUCCESSFUL or FAILED.) When this method returns, the stop has been requested, but may not be complete. If the backend is able to eventually stop the task, its status will become CANCELLED.
    • Backends that cancel tasks synchronously should document that.
    • The default implementation of BaseTaskBackend.cancel() would be to call self.get_result(result_id) (if the backend supports_get_result) to handle raising TaskResultDoesNotExist. Then it would unconditionally raise TaskNotStoppable(result_id).
  • BaseTaskBackend would also gain aabort() and acancel() as async versions of the above.
  • BaseTaskBackend.supports_abort: Whether the backend supports stopping Tasks which have started running. A backend that supports_abort must supports_cancel (which could be enforced in BaseTaskBackend.check()).
  • BaseTaskBackend.supports_cancel: Whether the backend supports stopping Tasks which have not started running.
  • TaskResult.abort() calls self.task.get_backend().abort(self.id) and likewise for aabort(), acancel(), and cancel(). Callers must call refresh() themselves if they want updated information.

In this API, the caller is indicating how much force it is willing to accept, and the backend can use up to that amount, but can use less. I have not currently differentiated graceful/coordinated stopping (e.g. Celery abort()) from forceful termination (e.g. Celery revoke(terminate=True)). If that is desirable, I can adjust the proposal to have a stop() for graceful termination. Alternatively, that could be added in the future, as abort() would remain the forceful termination.

DummyBackend should probably supports_abort (and supports_cancel). It would need to be changed from claiming “Task results will forever remain in the READY state.” to something like “unless aborted or cancelled”.

ImmediateBackend should support neither.

I’m using django-tasks-db and we implemented cancellation on top of it. Our specific implementation is different, but here is a proposal:

def cancel(self, result_id: str) -> None:
    from .models import DBTaskResult

    try:
        cancelled = DBTaskResult.objects.filter(
            id=result_id,
            status=TaskResultStatus.READY,
        ).update(
            status=TaskResultStatus.CANCELLED,
            finished_at=timezone.now(),
        )
    except ValidationError as e:
        raise TaskResultDoesNotExist(result_id) from e

    if cancelled:
        try:
            task_result = DBTaskResult.objects.get(id=result_id).task_result
        except (DBTaskResult.DoesNotExist, ImportError, SuspiciousOperation):
            logger.exception("Task id=%s cancelled, but its result could not be loaded", result_id)
        else:
            task_finished.send(sender=type(self), task_result=task_result)
        return

    status = (
        DBTaskResult.objects.filter(id=result_id)
        .values_list("status", flat=True)
        .first()
    )
    if status is None:
        raise TaskResultDoesNotExist(result_id)
    if status == TaskResultStatus.CANCELLED:
        return
    raise TaskNotStoppable(result_id)
2

Hi @rlaager, thanks for sharing this! Do you mind creating an issue on GitHub - django/new-features: New feature proposals for Django and third-party ecosystem · GitHub with these details?