What is the correct way to create an async task?

Using Django
1

Here’s why I ask:
A project I’m working on requires some long-running processes which can be easily done using asyncio. However, I’m not able to get them to run properly in an async view. After returning a response, the second a spawned task encounters an await keyword–it quits. No errors, no warning, nothing. It just stops executing. I believe this is because Django attempts to close the loop after the response is returned from the view.

The reason this causes issues for me is the client needs to parse the response for the next actions to take.

With that in mind, what is the proper way to spawn long-running tasks from an async view? Is it even possible?

To provide some example code, place this in a views.py file and properly configure the urls.py file:

async def http_call_async():
    for num in range(1, 200):
        await asyncio.sleep(1)
        print(num)

    # TODO: send email async
    print('done')

async def async_view(request):
    loop = asyncio.get_event_loop()
    loop.create_task(http_call_async())
    return HttpResponse("Non-blocking HTTP request")

Thanks for your time,
Jules

2

Hi Jules,

How are you running your project? runserver by default will run all code in a synchronous thread, so you need to use something else like Daphne to run you app locally.

3

Hi Tim,

Thanks for the response. I’m running the code in a production environment using a Gunicorn server with a Uvicorn worker. So it is running asynchronously as far as I can tell.

Here is the service configuration I am running on the server:

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=django
Group=www-data
WorkingDirectory=/srv/server
ExecStart=/srv/server/venv/bin/gunicorn \
        --access-logfile - \
        --workers 3 \
        --bind unix:/run/gunicorn.sock \
        --timeout 300 \
        --error-logfile /var/log/gunicorn/error.log \
        --access-logfile /var/log/gunicorn/access.log \
        --log-level debug \
        --capture-output \
        -k uvicorn.workers.UvicornWorker \
        webserver.asgi:application

I’m happy to provide any further information you require.

Jules

4

Have you confirmed that this works if http_call_async is shorter?

If these are very long running processes, I’d contemplate moving them to a background process such as Celery or Django Channels background workers.

5

My understanding is that that is correct.

Django itself is still by its nature, tied to the request/response cycle.

If you want a long-running process to continue after the view has finished, you’ll want to use something like Celery to run that process in a different thread / process.

The current functionality of an async view is to allow it to (potentially) gather data from multiple sources in parallel before preparing the response.

6

I do not believe the length of time http_call_async is actually the issue here. When the loop encounters await asyncio.sleep(1), it dies. It doesn’t print anything, nor complete execution of the function. This is evident because ‘done’ isn’t even printed to the console.

The only way I’ve gotten ‘done’ to print is if I wrap all of http_call_async in a try/finally block and put print('done') in finally like below:

async def http_call_async():
    try:
        for num in range(1, 200):
            await asyncio.sleep(1)
            print(num)
    finally:
        # TODO: send email async
        print('done')
7

I see. I was hoping to avoid Celery as it over-bloated for this use case in my opinion.

The only other option I was considering would be to spawn a separate, views.py–global asyncio loop independent of Django running in another thread. Then I could use an asyncio.Queue object to pipe what I need to those processes.

What are your thoughts on that work around? If you’d be against it, I’ll resort to using Celery.

8

(Emphasis added. Note that I don’t specifically recommend Celery here.)

It’s all a question of how reliable you want this background processing to be.

If it’s “fire & forget & hope for the best”, then yes, just about anything will do.

But, if you want to ensure it’s “Process once and only once” with a way of tracking failures and retries, then you’ll find that Celery is about as easy as it gets. That type of reliability come at a price - what you might call “bloat” (but I don’t).

9

Thank you for your suggestions and information. To you as well @CodenameTim; both of your insights have been very helpful.

A project I had worked on before encountered an issue similar to this and I think its solution will just about perfectly fit my use-case here as you encouraged @KenWhitesell. The solution for it was something like this:

secondary_loop = asyncio.new_event_loop()
secondary_thread = threading.Thread(
    target=secondary_loop.run_forever,
    name="Secondary Views Loop"
)
secondary_thread.start()

...

asyncio.run_coroutine_threadsafe(
    run_process_async(),
    secondary_loop
)

As much as Celery would offer for reliability, I just feel it’d be too bloated. By the by, I don’t mean bloated as a package itself, I just mean in terms of what I’m trying to accomplish.

I appreciate both of your help with this.
Jules

1 Like
10

Can I suggest looking at Huey or DjangoQ, which are like Celery; but specifically for these purposes, and are nice and lightweight. DjangoQ integrates in the Django admin as well in case that’s useful.

1 Like