async orm \ refreshing view /

Hi everyone! I’m sorry I’m a bit late for async-orm discussion

I have an idea how to make a version that supports both sync and async. The main idea is to extract all I/O into drivers. The code should not be allowed to do any I/O. Instead it should yield SQL, that is to be executed later somehow (in a sync or async way):

SQL = namedtuple("SQL", ["sql", "params"])

For example, as you know, we have a save method in models.Model. Let’s rewrite it in the following way:

    def save(...):
        yield from self.save_base(...)

Yes, a generator. What it yields? SQL instances, of course. A couple more edits should be made, of course.

def _save_table(...):
    yield from self._do_update(...)

def _update(self, values):
    yield from query.get_compiler(self.db).execute_sql(CURSOR)

def execute_sql(self, result_type=MULTI, chunked_fetch=False, chunk_size=GET_ITERATOR_CHUNK_SIZE):
        sql, params = self.as_sql()
    yield sql, params

Given that generators can return values, not only yield, this is a very effective way to work with existing codebase

Now, our models.Model is incompatible with the initial version (which is not a generator). How do we fix that? Easy: we should rename our class with a generator into AbstractModel, and then make

class Model(AbstractModel):

    def save(...):
        queries = super().save(...)

class AsyncModel(AbstractModel):
    async def save(...):
        queries = super().save(...)
        await async_driver.execute(queries)

What regarding lazy attributes: I’ve found out my point of view (among many others) has already been expressed in the Asynchronous ORM post

I consider that we should keep current syntax for the case when all objects have already been prefetched, that means, we do have them in our cache. When we do want to make a query, we should use another syntax, like this:

await obj.R("related_objects").all()

Now the most interesting part: if I ever wanted to go ahead with my proposal, then I wouldn’t actually need any approval from Django. The success of the project will be defined by how good the async version will be. Because that’s what’s actually missing from Django. For the sync usecase people will be able to use the traditional django (and will be suggested to do so, for some period of time).

But then, if the async version will be successful, then there are high chances that some day that will become the new version of django (because it will contain the sync version too). And in that case we probably want to make less breaking changes. How Django Foundation does usually treat such cases?

And also, please tell me what you think about my aproach/

I appreciate your idea - and it’s a decent basis for an async ORM - but the main design problem with making the Django ORM async is not changing the API significantly, but instead presenting a familiar interface and reusing as much of the existing code as possible (the ORM is over 50% of the code in Django by some metrics).

Thus, totally changing the way users call it is unfortunately a non-starter.

Thank you, Andrew. Not sure if it’s of any use but I developed this idea further in this gist (my manners suck too, so I tagged almost every developer in the python application space I knew). The API of the sync version is preserved 100%, and almost all the code is reused between the sync and async versions