How to set up background tasks with Heroku & AWS Lambda?

Using Django Deployment
1

Hey there :wave:

I’m currently building a Django app on Heroku. I’m now required to move some of the tasks that are currently running in my views to running them in background tasks (as Heroku has a 30-second timeout for requests, and often times these tasks would take longer).

The tasks are basically: scrape some data, run some analytics over the data, store data & analytics in DB, query DB and display it in frontend.

What I’ve tried so far is to move the task into an AWS Lambda function. The function works fine, but when I invoke the function using boto3 in my view, my view is still waiting for a response of the lambda function, therefore still running into the 30-second timeout.

I’ve done my research and identified a few different options to solve this, but would love to get some outside perspective on which one would make the most sense:

  1. Use Celery & Redis to manage the background tasks, and let the tasks be executed on AWS Lambda.
  2. Use Celery & Redis to manage the background tasks, but let the tasks be executed in a Python script on Heroku.
  3. Trying to solve it with asyncio in order to keep it leaner (not sure whether that specific case could be solved with asyncio, though?)
  4. Use Django channels.

I’m tending towards Celery & Redis with AWS Lambda, mainly because I haven’t used Celery & Redis yet and would like to learn it. But maybe in my scenario, this introduces unnecessary overhead and I’d be better advised to use something more simple?

I appreciate any type of feedback/help!

2

Hi,

There are two way to call a lambda function one of them blocks the client and the other one doesnt, you need to change the invocation type argument in the lambda client.

I don’t remember what value you need to use exactly, but you can Google it.

1 Like
3

Ah great, I didn’t know about that! Thanks for sharing, I’ll look it up :slight_smile: