How to use Django async on scraping and orm?

Using Django Async/Channels
1

I’m sure there are many options in this regard. So I want to concentrate on what is best.

I combined ORM save and API request under the same function is this wrong for async?

How can I set this function as async?:

def main():
	for product in inventories:
		id = product["id"]
		data = remote_api.products.get(id) #get information from remote server (type class)
		
		if data["status"] = "success": #If the request is successful, proceed to the orm save stage
			
			try:
				offer = Offer.objects.get(id=id)
			except ObjectDoesNotExist:
				offer = Offer(id=id)
			finally:
				offer.price = data["product_price"] #value to save
				offer.save()

Second Questions:
I need to set the rate limit to 5 when receiving data from the remote server. How can I handle this on the “for loop” side?

2

running on ASGI i would try something like:

from asgiref.sync import sync_to_async
import asyncio

async def main():
    async for product in inventories:  # 'async for' may be invalid, depending on what "inventories" is.
        id = product["id"]
        data = await remote_api.products.aget(id)  # Django>=4.1, async get method.
        # data = await sync_to_async(remote_api.products.get)(pk=id)  # if api calling function is not a coroutine. However it ought to be awaited.
        if data["status"] = "success":
            try:
                # offer = await sync_to_async(Offer.objects.get)(id=id)
                offer = await Offer.objects.aget(id=id)  # Django>=4.1
            except ObjectDoesNotExist:
                offer = Offer(id=id)
            finally:
                offer.price = data["product_price"]
                await offer.asave()  # Django>=4.1, async save() method.

alternatively you can specify synchronous calls in an extra function and call it async from within main() coroutine:

@sync_to_async
def fetch_data(id):
    return remote_api.products.get(id)

and within main():

...
id = product["id"]
data = await fetch_data(id)
...

Django Ninja async support documentation - further reading
Django async support documentation - further reading

I’m open to feedback on this.

1 Like