How to use Django ORM from Twisted event loop?

Using Django Async/Channels
1

Hi there, I am attempting to use the Django ORM from within a Twisted event loop. With previous versions of Python it was simple, set up the PYTHONPATH and DJANGO_SETTINGS_MODULE, call django.setup() and everything worked. Now with Python 3.9, Django 4.1.3, Twisted 22.10.0 I have all kinds of threading issues.

Currently I am just trying to do this:

from asgiref.sync import sync_to_async
from django.contrib.auth.models import User
async def register_user_client(self, username):
u = User.objects.get(username=username)

But this doesn’t work. I get “You cannot call this from an async context - use a thread or sync_to_async.”

So, fine, I try to introduce sync_to_async:

from asgiref.sync import sync_to_async
from django.contrib.auth.models import User
async def register_user_client(self, username):
u = sync_to_async(User.objects.get, thread_sensitive=True)(username=username)

This gives error “twisted/internet/defer.py:1697: builtins.RuntimeWarning: coroutine ‘SyncToAsync.call’ was never awaited”

So I add an await:

from asgiref.sync import sync_to_async
from django.contrib.auth.models import User
async def register_user_client(self, username):
u = await sync_to_async(User.objects.get, thread_sensitive=True)(username=username)

To this I get “builtins.RuntimeError: await wasn’t used with future”

I also tried a seperate method with the sync_to_async decorator:

from asgiref.sync import sync_to_async
from django.contrib.auth.models import User
@sync_to_async
def get_user(self, username):
u = User.objects.get(username=username)
return u

async def register_user_client(self, username):
u = self.get_user(username)

I get “twisted/internet/defer.py:1697: builtins.RuntimeWarning: coroutine ‘SyncToAsync.call’ was never awaited”

So I try with an await:

from asgiref.sync import sync_to_async
from django.contrib.auth.models import User
@sync_to_async
def get_user(self, username):
u = User.objects.get(username=username)
return u

async def register_user_client(self, username):
u = await self.get_user(username)

Again I get “builtins.RuntimeError: await wasn’t used with future”

I also tried an asychronous support call:

from asgiref.sync import sync_to_async
from django.contrib.auth.models import User
async def register_user_client(self, username):
u = await User.objects.aget(username=username)

To this I get “builtins.RuntimeError: await wasn’t used with future”

If I take out the await:

from asgiref.sync import sync_to_async
from django.contrib.auth.models import User
async def register_user_client(self, username):
u = User.objects.aget(username=username)

I get “twisted/internet/defer.py:1697: builtins.RuntimeWarning: coroutine ‘QuerySet.aget’ was never awaited”

I have also tried these combinations without an ‘async’ method with similar results, and of course you can’t use an await without async.

Clearly I have been working at this for a long time. Can anyone please tell me what I’m doing wrong? I’m really hoping the answer is not that it is impossible to use Django ORM from within a Twisted run loop.

Thank you.

2

I don’t have an answer to your specific question.

However, I’m fairly confident that it can be done because Channels appears to use twisted in Daphne. (See daphne/server.py at main · django/daphne · GitHub)

You might want to take a detailed look at how Daphne works and see if its logic can be adapted to your needs.

3

I was able to figure out a solution to this, but it is ugly. If anyone has a better, more elegant solution please let me know. The following works:

from twisted.internet import threads, defer
from django.contrib.auth.models import User

def _get_user(self, username):
u = User.objects.get(username=username)
return u

@defer.inlineCallbacks
def get_user(self, username):
u = yield threads.deferToThread(self._get_user,username)
print("USER:",u)

This has a massive downside, however. You have to make sure the _get_user method returns actual result data and not a queryset because once you are out of the thread you can’t access the database for more data. In other words you need to eager load everything.

As I said, if anyone has a more elegant way to do this please let me know.