Background Tasks using Celery based on model instances

Using Django Deployment
1

Hey everyone,

I’ve been thinking about a project and want to use Django for it. I have some Python background and worked through Django tutorials before, so I have an idea how to set-up a Django project and make it do things.

For my current project, however, I want to make use of Celery tasks to run stuff regularly in the background. I have found the documentation and several good examples on how to set up and use Celery. Those examples usually concern primarily maintenance tasks, though.

More concretely, I want to have monitoring tasks running in the background and writing results in the database. Consider the following (simplified) models:

# models.py
from django.db import models

class BaseWatcher(models.Model):
    name = models.CharField(max_length=255)

    class Meta:
         abstract = True

class HTTPWatcher(BaseWatcher):
    target_uri = models.CharField(max_length=255)
    interval = models.DurationField()
    last_status_code = models.IntegerField()

When adding an instance of HTTPWatcher (e.g. through the Admin UI), I know want to create a periodic Celery tasks that does some simple HTTP request logic and put the resulting status code into the database for this instance.

Where would be the best place to put the code for the corresponding task? The models.py does not seem the proper place for it.

If someone can provide me with a pointer to some fitting example or introduction, that’d be appreciated!

Thanks & best regards
Christopher

2

If by periodic you mean that it runs on a schedule (such as once per hour), then I’d recommend making it a custom django-admin command that can be run either as a cron job or from Celery Beats.

On the other hand, if you mean periodic in that some event within your system triggers it to run, then I would create it as a normal Celery task - which in my case means it would reside in my tasks.py file.

3

Hi Ken,

thank you for your speedy response! Not sure, if the django-admin approach would work for me.

To elaborate on the example above:
A user would be able to create a new HTTPWatcher – based on my definition in models.py – and provide a target URI (e.g. google.com) and a specific interval (e.g. 60 minutes). Along with the object being in the database, I need to create a Celery task (i.e. for each instance created by the User) that sends the request to google.com every 60 minutes and writes the status code back to the object.

If I understand you correctly, the custom admin command would require me to do something through the command line in order to create the cron job or Celery task.

Best
Christopher

4

I think it could, but will address this later.

Not necessarily - that’s where Celery Beat comes into play.

You could create your watcher as a regular Celery task, then schedule it to run every hour using Beat. The Beat task you create can pass parameters to the task, so you can use the same task for everything you’re watching.

(Note: Be aware that using celery beat adds another process to your deployment. You’ll have a task for beat in addition to your process for celery.)

Now, the alternative that I was thinking of for the cron job, would be to create your watcher task as a command-line command. It would read the list of entities to be watched from the database and test them each time it runs, saving the results back to the database for the corresponding URL.

Adding and removing entries to be watched is then a database change, and does not require changes to the task itself. If you’re only using beat for this purpose, then it’s probably easier overall to create this as a single cron job and not add the overhead of beat.

5

Hi,

I think you can achieve this as Ken suggested you, I suggest you to put an eye on this package

It allowed me to do something pretty similar.

6

Hi Ken, hi Daniel,

thank you for following up and your explanations.

I dove deeper into django-celery-beat and was able to set up what I was aiming at. In particular, I now have django-celery-beat running and when saving the model, I create a periodic task (actually using the post_save signal). The task itself now lives in tasks.py.

I.e. I don’t use custom commands now, because it seemed to me that it’s not necessary for my use case.

Might be able to post a quick example later if it is of interest to anyone.

Thank you & best regards
Christopher

1 Like