DEP0009/ORM and backward-compatibility

I’m working on a PoC for DEP0009, and I a little unclear on the backward-compatibility approach.

Currently, it is possible to await an async method (eg .aget()) with a sync-only DatabaseWrapper, and it will call the correspondent synchronous method wrapped in a sync_to_async. In other words, currently this is possible:

from my_app.models import MyModel

my_obj = await MyModel.objects.aget(pk=1)

DEP0009 states:

Backwards compatibility means we must let users access connections from any random code whenever they like, but we will only allow this for synchronous code; we will enforce that code is wrapped in a “connection context” if it is asynchronous, from day one.

My understanding is that this is the DEP’s intended behavior:

from my_app.models import MyModel

my_obj = await MyModel.objects.aget(pk=1)
### raises some NotInAsyncException

Wouldn’t that break backward-compatibility? Should PEP0009 be revised to include a migration plan?

Hi @fcurella :wave:

First off, thanks for picking this up. It’s an exciting next step! :gift:

So, I don’t know exactly what was in mind with a “connection context” but, my understanding was that we already did this, in that if you try and make a (sync) DB call from an async context, without going via sync_to_async() you’ll get a SynchronousOnlyOperation error, as per the Async safety section of the docs.

I’m not sure what more is needed or intended there? :thinking:

If we have a genuinely async query path then that’s already covered — you can’t use await outside of an async def function — so again we’re good no?

Likely I’m missing something! :slightly_smiling_face:

Consider this example:

from my_app.models import MyModel


async def my_view(request):
    my_obj = await MyModel.objects.aget(pk=1)
    ...

Currently, such a view doesn’t raise any exception.

PEP009 defines an async contextmanager, new_connection. ORM calls are supposed to be allowed only inside the connection context that new_connection creates:

from django.db import new_connection
from my_app.models import MyModel


async def my_view(request):
    async with new_connection:
        my_obj = await MyModel.objects.aget(pk=1)
    ...

The ORM call in the first version of the view won’t have a connection context to work with, and will raise an error.

I guess we could wrap async view with new_connection by default (haven’t look into it), but I’m wondering about other code that’s not necessarily inside a view (management commands come to mind)

And presumably we do still need something along this line because (if I’m not just inventing things) if the underlying code is async we’d need an async connection, yes?

Ok… thank you for your patience, I think I’m with you.

Do you have any thought for a migration?
It might be that we just have to add the break there. (That’s not optimal clearly…) :thinking:

We could detect the situation and issue a warning before a later version throws an exception? That at least partially mitigates the pain of this change.

Quoting from the DEP:

In an asynchronous world - where all coroutines run on the same underlying Python thread - this goes beyond being annoying to being outright dangerous. Without any extra safety, a user calling the ORM the way they do today would risk cross-thread pollution of the connection objects.

I’m not terribly familiar with how the ORM handles connections, but is another alternative to clean up the cross-thread pollution?

This idea might have some merit. As I understand it (currently afk) database connections are cleaned up at the end of every request anyway by a signal hook. There’s some nuance here around how “forever” connections can stick around but it might work.

One option would to fall back to the current sync_to_async behavior when we don’t have an async connection, emitting a PendingDeprecationWarning.

Something like:

class QuerySet:
    ....

    async def aexists(self):
        try:
            return await <async_implentation>
        except MissingAsyncConnection:
            warnings.warn("...")
            return await sync_to_async(self.exists)()
1 Like

That seems perfectly reasonable.

  • Introduce context manager to get an async connection.
  • Deprecate using async ORM methods without the context manager.
1 Like

For the benefit of others, like me, who didn’t remember what DEP 0009 is, it’s a Django Enhancement Proposal (DEP) for Async-capable Django

Full doc at deps/accepted/0009-async.rst at main · django/deps · GitHub