Hi guys,
I’ve been facing a pretty perplexing issue regarding web scraping w/ a django web server.
You see, I’ve coded a successful async method that uses aiohttp to scrape amazon.com. When I run this async method in a regular main() method, it seems to work. However, whenever I run it using django, the second request always seems to get caught by amazon.
I’m using the aiohttp client session getter referenced here:
Can anyone help me out here? Thanks!
Can you post your code? Which part exactly does not work?
Alright, lemme try and give a clearer explanation(sorry about that).
Basically, I am using a django server to scrape amazon.
I have an async function that takes a link and scrapes the amazon website for the ingredients of that page.
This works 100% of the time when I run these functions together using asyncio.gather().
However, when I try to run the asyncio.gather() method within a django view, the first link will always work, but the second link always fails(now, it simply returns a “not found” page even though I literally copy-pasted the link from my browser).
My question is that if there’s any ASGI configuration that could help fix this. Thanks!
But what is your current code? What is your current configuration?