Fielding ideas/ products to use for background workers in Django outside of celery. We’ve had a lot of issues with celery and want a more managed system. We’ve migrated some of our async processing to Temporal but even then we have to manage the cloud compute for the workers. Has anyone used a more managed service for their Django backend? I have seen Inngest but am unsure if it would resolve our issues.
Welcome @cole-loma !
I’m curious, what issues are you encountering with celery, and what specific management features are you looking for?
I ask, because we’ve never encountered any intrinsic instabilities when using Celery and find it to be extremely manageable from any number of different angles. (Granted, there is some degree of management that requires tools external to Celery itself, but they’re generally the types of things you’d want to manage outside the processes themselves.)
Hey @KenWhitesell , I joined this team mid-migration from Celery to Temporal so the original motivation is not entirely clear to me, however, I do know that what we are looking for is essentially a service that we can call a function from Django and it runs in the background without us managing queues, scaling, workers, etc.
The closest thing I’ve seen to this is Inngest but I have yet to test this out and was curious what people out there are using.
Oh, ok. So what you’re really looking to do is to outsource the process execution, not to manage your own.
How are you currently hosting your django backend? If you’re talking about not wanting to manage compute at all, then presumably you are already hosting the backend on some paas somewhere. If it’s on AWS, you can look at running adhoc jobs using aws lambdas. Otherwise pretty much any paas platform is going to have a concept of a “function” or a “job” that you can trigger. It will be ideal if you are hosting your job orchestration in the same place as your web application.
To answer your question, I have my django app hosted in kubernetes. We use celery for high throughput tasks that can be run on the fixed instance sizes of the celery workers. Aside from that, kubernetes can be used as a “non durable” job orchestrator. So I also use kubernetes to run low throughput batch jobs that have variable compute constraints for my django project. I wouldn’t suggest using kubernetes since you were talking about not managing your compute, but I wanted to share.