Looks like their secret sauce is the libs they’re using
From their readme:
- SQLite (requires aiosqlite)
- PostgreSQL (requires asyncpg)
- MySQL (requires aiomysql)
So its Not relevant ?
It’s relevant IMO. I think staring at the design of those libs (or even using them as dependencies) is a viable path forward for Django’s ORM.
But I’m not a part of the Django team, so it’d be best for @carltongibson to answer that.
It’s all relevant. Looking at what’s out there, and seeing how they’re doing it.
It would already be feasible to pass the SQL with parameters generated by the ORM to one of the async libraries and then implement an iterator to return model instances. … (e.g Tom Christie’s databases package uses SQLAlchemy to generate the SQL but you could slot the ORMs SQL in there directly.)
The issues come up when you have lazy related lookups, which would trigger another query. How are we going to handle those (see discussion above) That’s a harder topic. But not one you couldn’t play with.
1 Like
Right - I think taking inspiration from other ORMs is fine, but remember that the main problem with adding async into the Django ORM is doing it in a way that integrates with the existing design patterns and code.
If we were to design a brand new, second ORM to achieve async - well, that would be a lot easier to make properly async, but it would kind of defeat the point.
3 Likes
How andrew, how you think switching to psychopg3 would ease the effort? psycopg 3.0.1 documentation itwas released recently. also is django going to switch over it over current v2.x?
@andrewgodwin So after seeing that Django is likely not gonna make async orm into version 4.x. I made my own package to make my life easier and hopefully a lot of others developers out there. Repo:
Disclaimer: Async managers and querysets are only wrappers around the synchronous orm:
The package includes:
- Async model mixin
s(fully typed),asgimod.mixins. - Async managers and querysets (fully typed),
asgimod.db. - Typed
sync_to_asyncandasync_to_syncwrappers,asgimod.sync.
Documentation and API references are in the README.
Testcases and examples are in testapp.test module at the root.
- Does this support foreign relation access: YES.
- Does this allow queryset chaining: YES.
- Does this allow queryset iterating, slicing and indexing: YES.
- Does this affect default model manager functionality: NO, because it’s on another classproperty
aobjects. - Is everything TYPED: YES, with the only exception of function parameters specification on Python<3.10 since PEP 612 is being released on 3.10.
Hopefully this can be helpful for the Django community and contributors, some inspiration on the async orm or maybe snap this in into your official docs to help other devs.
The docs on README may be confusing, I was lazy to write html to make it more appearing. Testcases only covers the most used queries and methods, some rare methods are not covered, if someone want to PR the uncovered test cases, feel free to do so.
Also a quick explanation of WHY I chose to not extend models.Manager and add the async equivalent methods using the a prefix as I proposed earlier in this forum: The manager has methods that returns querysets, and querysets can chain more methods and to build up the internal query. For example:
await Model.objects.afilter(a=b).aorder_by("a").areverse().aeval()
It makes absolutely no sense to call repetitively a prefixed methods when you can choose the correct manager to do the job.
await Model.aobjects.filter(a=b).order_by("a").reverse().eval()
It also wouldn’t break consistencies because aobjects is at the same level of asave and adelete, which by consistencies the foreign relation access, I chose to also use the a prefix.
This package DOES NOT modify any predefined models of Django itself, meaning that the default User model, GeoDjango models, Oracles models will need to be extended with the mixin somewhere.
1 Like
@andrewgodwin I have tested this package on my own production projects, at least for the queries the project is using. Do you have any opinions on this ? Maybe I should do a separated post dedicated for this package.
I happened to stumble upon another async Python ORM today and figured it’d be worth linking here for reference.
Is this effort still alive? The original release target for this has long gone. Is there any update on timing?
I am not able to work on it as long as the pandemic is ongoing, since pretty much all my mental effort is being used up by that. Others are welcome to continue the work, or hopefully I’ll get back to it… in future?
2 Likes
I like this approach.
I am not very familiar with async, but it comes to my mind another syntax version
Model.objects.filter(a=b).order_by("a").reverse().eval().dasync()
it is more readable at a first glance, and dasync() (from Django async) to inform that it returns async version, equivalent of
await Model.aobjects.filter(a=b).order_by(“a”).reverse().eval()
Maybe it was considered at the beginning of the project.
Yep, I considered that when making asgimod, and it was originally .aeval, but this was dropped because I wanted ForeignKey (and other relations) access, for example a model has a OneToOneField other, o.other.aeval() doesn’t work because o.other would already return the object and raise exception. So all in all it was a matter of code consistency.
Another reason was that I would need to monkey patch a custom version of the model Manager class, which is something that needs to be avoided.
Another reason was that I would need to monkey patch a custom version of the model
Managerclass, which is something that needs to be avoided.
Why does monkey patching of model manager need to be avoided?
It’s a personal opinion, implicitely changing the behavior of the code of another module comes at the very last in my list. Others may think otherwise, which is respected aswell.
As I understand async, it is usefull with time consuming i/o operations. Processing Model.objects.filter(a=b).order_by(“a”).reverse().eval() is not i/o operation, it is CPU/memory operation. And until it sends statement to data base it is quite quick vs db operations. So one place where it is worth apply async is connecting to db, sending statement and waiting for result from db.
It is worth to using profiler to see how much time takes every part of that. We need to remember that every await-async is context switching time consuming and resources (cpu, memory) consuming as well.
I was wondering if django orm would work as async orm (async data base operations only) do we need bother about sync version longer? Even if user wouldn’t use async “frontend”, orm could work asynchronously if python version supports it. It would just send database statement asynchronously.
For backwards compatibility reasons. If the old ORM is immediately dropped in favor of async, then all existing code breaks. For example, any package on PyPI that makes ORM calls would immediately be broken. Same with existing Django projects.
Based on what I’ve seen over the years, the Django team has a heavy priority on long-term backwards compatibility. So in this case, there’s an active effort to avoid breaking a large number of packages.
I personally think Python async was a big enough language milestone to make breaking changes within Django, similar to how Python 2 to 3 broke the ecosystem. Since the Python foundation has said Python 4 will likely never come, that scale of breaking changes can only occur as a deliberate decision on Django’s side.
But deliberately making catastrophic/cascading breaking changes is generally hard to justify.
(Agreed:) Not gonna happen. Way too much breakage.
Anything introduced has to be backward-compatible.
(Agreed:) The Python 2 to 3 transition was a long painful (and unintentionally severe) break that lasted well over a decade. Not something I’m looking to repeat as a part-time maintainer.
+1