Hi, I recently enjoyed listening to Carlton Gibson’s Django’s Async Story podcast, and exploring the Django’s async feature.
One thing I noticed is that accessing database (using ORM’s etc) is still marked as @async_unsafe, which raises SynchronousOnlyOperation error when you are in the event loop. It is also mentioned in the documentation.
However, I was wondering if there are actually still async-unsafe.
When I tracked down the Git history, the @async_unsafe was first added in this PR in 2019:
If I am understanding correctly, at that time, the database connection was a threading.local(), which blocked database operations be performed when there is a running event loop since the database connection should not be shared between different tasks.
And in 2023, I found a PR in asgiref, changing the asgiref.local.Local() to context vars when there is a running event loop:
with that change, the database connection is not shared between different tasks since they would have different context vars per task.
So my question is, is database operations still async-unsafe? Are there other risks that could happen when accessing the connection inside the event loop?
I am new to the django internals, so sorry if my question is something too obvious! Thanks!
Hi @ryanking13. Yes, focusing just on Postgres, connections still use sync cursors, so direct access needs to go via the sync_to_async helpers. The ORM’s async API handles this for you, so you should treat it as an implementation detail in most cases.
FWIW pyscopg v3 offers async cursors, but they’re still tied to a serialised connection to the database. (Psycopg takes a lock to enforce the synchronous ordering of queries.) So, under the hood it’s just a question of exactly where you make this switch.
I’m glad you enjoyed the podcast episode. I’ll link it here in case others haven’t seen it:
@carltongibson Thanks for the explanation! I have a follow-up question about this part:
connections still use sync cursors, so direct access needs to go via the sync_to_async helpers
I am curious which one is the main concern between:
(Performance) Blocking the other task from running when using database / cursor.
(Security) Data corruption because of the sharing global state.
Django’s Asynchronous support documentation mostly mentions data corruption, but the sync cursor problem you mentioned sounds like it is about the performance.
To provide more context, we are working on supporting Django in Cloudflare’s Python Workers, which runs Python on top of JavaScript runtime, which is event-loop-based, and single threaded.
We are curious if there is any security risks in using async-unsafe part inside the event loop and we are less worried about sync cursor blocking other requests.
OK, so… it’s ultimately about connections, and the transaction state on them. Django’s connection model is essentially connection-per-thread, and you need to be careful to make sure queries are targeting the right connection, and transaction. (sync_to_async goes to a lot of work to do this for you.) So, in general, it’s both about performance and safety.
But…
Right… so… could I run my queries on the current thread, blocking the event loop. Yeah, sure, probably. (If I was careful about it — and in a single-threaded environment that might just already be the case. )
If you’re sure you’re in this circumstance you can set the DJANGO_ALLOW_ASYNC_UNSAFE environment variable to bypass the @async_unsafe checks.
Yeah, we are a pretty weird runtime with different assumptions about event loops and threads
Yes, every concurrent request runs in the same single thread, and coroutines yield between the await points. Since we are single-threaded, loop.run_in_executor is a no-op, and the task passed into that function always runs in the same thread/task.
There’s a big difference between an OS lock plus a thread executor, and an async lock that stays purely in userspace.
An OS lock under contention leads to a syscall. A thread executor adds at least two context switches. The async lock has none of these problems: no syscall, no context switching, no convoy effect. The cost is real, and it’s measurable.
Yeah, I can imagine it working fine. You’ll need to watch out for interleaving transactions, so disabling atomic requests for example, if you have concurrent requests, but otherwise, I’m sure there will be wiggles, but not an in principle issue… — it would be interesting to see how you get on.
)
