Async Django ORM (django-async-backend)

Django Internals Async
1

The result of long and hard work on django-async-backend :relieved_face: What did we get? What didn’t we get? What problems do we have, and where is this going?

Short context

Django has several PRs trying to implement a real async ORM. They take different approaches, and some of them have hundreds of comments, but they all share one thing: none of them will ever get merged. As a result, a separate project was created that implements an async Django database backend: django-async-backend. That wasn’t enough for tests, so we went further and started implementing an async ORM.

What did we get?

For now, we have:

  • A pure async Postgres connector
from django_async_backend.db import async_connections
from django_async_backend.db.transaction import async_atomic

async def main() -> None:
    connection = async_connections["default"]

    async with async_atomic():
        async with await connection.cursor() as cursor:
            await cursor.execute(
                "INSERT INTO app_book (name) VALUES (%s) RETURNING id",
                ["Django"],
            )
            (book_id,) = await cursor.fetchone()

            await connection.on_commit(lambda: callback(book_id))
  • An async ORM that’s ready (95%), plus transactions, polling, etc.
async def main() -> None:
    connection = async_connections["default"]

    async with async_atomic():
        book = await Book.async_objects.acreate(name="Django")
        await connection.on_commit(lambda: callback(book.pk))
        
        ...
        

        book.name = "Django Async"
        await book.async_save(update_fields=["name"])
  • Unit test utilities
from django_async_backend.test import AsyncioTestCase

class MyAsyncTests(AsyncioTestCase):
    async defasyncSetUp(self):
        # Setup code

    async def asyncTearDown(self):
        # Cleanup code

    async def test_something(self):
        # Your async test logic
        await do_async_stuff()

It works. It’s tested. It’s used in real projects. You can check the full list of ORM methods here: ORM queryset methods.

What didn’t we get?

We didn’t keep Django’s default async interface:

  • atomic → async_atomic
  • asave → async_save
  • adelete → async_delete
  • User.objects → User.async_objects

There are two reasons for this:

  • Django’s async ORM interface isn’t designed to be pluggable with a real async backend and real async queries, because it’s inextricably tied to the sync connection. A sync cursor and an async cursor can’t share the same connection. You have either a sync one or an async one, and you can’t use one for both. As a result, any project that uses both ORM versions (which is 99.9% of async Django projects, since Django doesn’t support async transactions) would break.
  • Django already has known bugs related to fallback for asave / adelete and similar methods. Keeping the same style means keeping the same problems and generating tons of bugs for the end user.

What problems do we have?

Problem #1 is that Django’s async ORM interface isn’t pluggable, and that comes at a cost. To explain it, I need to describe a bit of the under-the-hood implementation. The connector part is hand-written, but the ORM part is auto-generated from the original Django codebase (to put it simply, it converts def → async def and adds await at each IO point).

It would be an understatement to say this isn’t an elegant solution. So why do it this way?

There’s no option to add it to Django core, so three options are left. The first is a hand-written, separate ORM and connector. Django ships hundreds of commits to the ORM part in every major release (new features, optimizations, bug fixes, security issues). Lots of people write patches, review them, and so on. A separate ORM means the maintainers (1-2 people) would have to do everything the Django community already does (alone).

The second option is to inherit from Django classes, reuse as much as possible, and rewrite only the parts with IO. This looks better, but it’s easy to break. In some commit, a maintainer adds new IO and it breaks everything. There’s no way to prevent that.

Auto-generation has its own disadvantages, but it gives us:

  • Control of the codebase and the ability to prevent issues (just cover all the code with tests and you get a guarantee that there’s no missed await).
  • A clear flow for applying and validating changes.

You can see what an upgrade looks like here: PR #74. It’s the upgrade from Django 6.0 to 6.1. It contains 60 commits. There were big changes related to deleting objects and fetch_mode, and it wasn’t too painful to update. Yes, it took a few evenings, but it feels feasible as a once-every-6-months activity.

The algorithm:

  • Update to a commit.
  • Add tests to reach 100% coverage.
  • Fix async/await if a test fails.

I can’t even imagine how many days it would take to keep this up to date without auto-generation.

I don’t like this solution (the one I’d actually like would require changes in Django core), but it gives us the ability to maintain the project and make it reliable.

Where is this going?

Here’s what we’ll do:

  • A few things in the ORM aren’t ready yet, for example prefetch_related, araw, etc. These are low priority, but we’ll implement them, if only to be able to say the ORM is 100% ready. :sweat_smile:
  • Increase test coverage to 100% (we don’t need great tests for everything, but we need to make all the code runnable so we can catch cases with a missed await).
  • Keep in sync with Django releases.

Things I’d be happy to see in Django:

Django’s ORM is one of the biggest and hardest parts of the Django codebase, so moving the async version out into a third-party package sounds a bit crazy :slightly_smiling_face:. The best option is to have it as part of Django core, or at least make it pluggable. (I know the core devs think sync_to_async is fantastic and enough, but I think end users need to have a choice.)

Steps I’d take:

  • Provide base abstractions for the key parts: BaseAsyncDatabaseWrapper, BaseAsyncConnectionHandler.
  • Make the async connection pluggable. Provide an abstract AsyncConnectionHandler and utils for working with it. Integrate atomic and other utils with it (ideally provide a way to have a separate backend for async and sync connection handlers, rather than a single mixed one like it is now, and not only for the database).
  • Make the async ORM pluggable. Provide an async manager with abstract methods. Third-party libraries can use this contract to interact with Django and stay interchangeable. (This solves the asave / adelete problem.)
  • Make the existing ORM more async-friendly. Solve the problems with contenttypes and updates (or at least prepare workarounds).
  • Introduce an AsyncioTestCase with all the existing Django features (settings_override, client, etc.) but without async_to_sync and the database (delegate those to third-party packages).
  • Rewrite QuerySet (and some model methods) as async and generate the sync version with unasync, like Psycopg3 and others do.
  • Implement an AsyncDatabaseWrapper for Postgres :slightly_smiling_face:.

This way, Django can step by step take over the stable parts of third-party packages, provide a contract for them, and eventually implement an async ORM, at least for Postgres.

It would need a push from the community and willingness from the core devs, at least to review it. But either way, it’ll take years. So if you need an async ORM, or you’re interested in it, please join.

3 Likes