Complex dependencies and relationships between Celery tasks

As mentioned in another thread, I’m working on an integration between my Django REST-based LMS and Google Classroom.

Using signals, when certain actions happen in my LMS, I dispatch specific actions from an “integration class”, which ultimately fire API calls to the Classroom API.

I run the integration class methods as Celery tasks. I didn’t want to have to code the Celery-task-related logic inside of my integration class, in order to separate the business logic with the logic relating to executing the methods, so I made a task that receives the method name and the kwargs and runs the method with the given kwargs.

Since the integration class’ methods accept model instances as arguments, I also perform “(un)marshalling” of the arguments by turning them to pk’s and querying for them inside of the task:

dispatching function sai_evo_backend/ at classroom_integration · Evo-Learning-project/sai_evo_backend · GitHub

task sai_evo_backend/ at classroom_integration · Evo-Learning-project/sai_evo_backend · GitHub

Here’s the issue:

some tasks rely on other tasks having successfully run before; however, those tasks aren’t run together.

I’ll give an example: when a student enrolls in a course in my LMS, they are also enrolled in the paired course on Classroom. Later, when a student participates in an exam in my LMS, which is paired with a courseWork item on Classroom, their studentSubmission is updated to contain a link attachment linking to the participation on my LMS.

The problem is: the studentSubmission might not exist if the user isn’t enrolled in the Classroom course (Classroom automatically creates a submission for each enrolled student and doesn’t allow direct creation of submissions).

So, in some way, the task that updates the studentSubmission depends the task that enrolls the students running successfully. So here’s the first issue.

Second thing: let’s suppose that the enrollment task fails, and due to this, the studentSubmission updating task also fails. If at a later moment I am able to successfully enroll the student on Classroom, I would like to be able to retry the submission updating task.

How would you tackle these issues?