SynchronousOnlyOperation error when running gunicorn/gevent and using async_to_sync

Using Django Async/Channels
1

Hi there!

We’ve been running into a recurring, confusing issue when using Django in conjunction with gunicorn (gevent) and channels. I have simplified the setup to this very simple code that is causing a problem:

View:

from django.views.generic import View
from django.http import HttpResponse
from channels.layers import get_channel_layer

def send_message():
    layer = get_channel_layer()
    async_to_sync(layer.group_send)("test_group", "test_message")

class HomeView(View):
    def get(self, request):
        # Send a message via channels
        send_message()
        return HttpResponse("")

So: a simple view that is just pushing a message to a channel layer every time it is hit.

We then run Django with:

gunicorn "myproject.wsgi:application" --bind "0.0.0.0" --workers 4 --worker-class gevent

Now, if we hit the HomeView with just a few dozen concurrent requests, we start running into this error when Django tries to query the database (in this case from the session middleware):

  File "/usr/local/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1880, in execute_sql
    with self.connection.cursor() as cursor:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/utils/asyncio.py", line 24, in inner
    raise SynchronousOnlyOperation(message)
django.core.exceptions.SynchronousOnlyOperation: You cannot call this from an async context - use a thread or sync_to_async.

I have tried using force_new_loop=True in async_to_sync, and that makes no difference. So something in this example seems to be initialising an event loop in the main thread, which then causes the async_unsafe check to raise an error.

As far as I can tell, nothing in here should be initialising the event loop other than the call to async_to_sync, so I’m very confused about what is going on. Is this some interaction between gunicorn/gevent and Django? What is the correct way to push to a channel layer with this setup without creating an async-unsafe context in the main thread?

2

Did you find the reason? I’m facing the same issue

3

Welcome @Enirsa !

I can’t speak for the original author, but I can tell you that when I need to write to a channel from within a synchronous instance of Django, I don’t use the Channels library. I use the regular Python redis library and write directly to the channel. (It works extremely well for me.)

1 Like
4

@Enirsa I was not able to identify the root cause. Ended up doing more or less the same thing as what Ken has suggested below - i.e., pushing directly to the channel layer instead of using async_to_syncwith the channels helper. I do think there is bug somewhere in the stack, but didn’t have time to track it all the way down - it’s also hard to debug given that it only happens after a number of requests have been served.

1 Like
5

Thanks, Ken!

For my future fellows with the same problem, here’s the fully sync workaround:

settings.py:

CHANNEL_LAYERS = {
    'default': {
        'BACKEND': 'channels_redis.pubsub.RedisPubSubChannelLayer',  # must be Pub-Sub
        'CONFIG': {
            'hosts': [REDIS_URL],
            'prefix': REDIS_PREFIX,
            'serializer_format': 'json',
        },
    },
}

Emitting events from sync code (views, models, signals etc.) using Redis client directly:

import json

from django.conf import settings
from redis import Redis


def emit_some_event(obj: SomeModel) -> int:
    client = Redis.from_url(settings.REDIS_URL)
    group = 'some_group'
    channel = f'{settings.REDIS_PREFIX}__group__{group}'
    event = {
        'type': 'some_event',
        'obj_id': obj.id,
    }
    result = client.publish(channel, json.dumps(event))

    return result

Websocket consumers can be kept as is.