Asyncio for Web Scraping: Concurrent Requests Done Right
TL;DR
asyncio improves throughput for many independent network waits; it does not make HTML parsing faster or grant permission to increase a site's request rate.
Reuse one aiohttp.ClientSession, set total and connection timeouts, and bound both global and per-host concurrency.
A semaphore limits simultaneous tasks, not requests per second. Add explicit per-host pacing when the source requires it.
Retry only transient failures such as 429, 502, 503, and 504, with a bounded delay and Retry-After handling; do not blindly retry every 4xx response.
For JavaScript-rendered sites or recursive discovery, call a managed rendering and Crawl API instead of expecting aiohttp to execute JavaScript.
When Should You Use Asyncio for Web Scraping?
Use asyncio when the scraper spends most of its time waiting for many permitted HTTP responses. One worker can manage multiple in-flight requests without a thread per request. The benefit depends on source latency, allowed request rate, parsing cost, and downstream storage. If the source permits only one request every few seconds, an asynchronous client cannot legitimately turn it into a high-rate crawl. The Nstdata Crawl product is a managed option for the acquisition and conversion steps discussed here.
The Python asyncio task documentation explains how coroutines are scheduled. The recommends reusing a session rather than creating one per request. A session keeps a connection pool and shared configuration. Use this tutorial only for sources you are authorized to collect.
Create a short allowlist of approved URLs, note each host's permitted request rate, and choose a contactable User-Agent. For Nstdata Crawl, obtain an account and API key; the documented authentication header is x-api-key. You do not need proxy Host, Port, Username, Password, or Protocol values for a standard managed Crawl request because Nstdata handles proxy routing.
How Does Concurrency Differ from Rate Limiting?
Concurrency is the number of requests in flight at once. Rate is how often new requests begin. A semaphore of three may produce three requests in the same instant, then another three as soon as responses finish. That can violate a source's desired pacing even though only three requests are active. Use a per-host start interval or token bucket when a host requires a maximum rate.
Set limits at three levels: the total number of active tasks, the connection pool's limit_per_host, and the per-host start interval. Keep them conservative during a pilot and adjust based on source policy, 429 responses, latency, and error rate. The example below uses one second between starts per host and at most three connections per host; these are illustrative settings, not a universal safe rate.
Step 2. Run python async_scrape.py and inspect each result or exception. A successful example.com response should contain its document title; the script does not claim that every target page has a <title> element.
Step 3. Add persistence only after validating the extracted fields. Store URL, fetch time, parser version, and rejection reason. Replace print with an idempotent write keyed to a canonical URL or source ID.
Limit: The Retry-After helper above handles numeric seconds. An HTTP-date value needs a date parser before production use. The example also holds the per-host pacing lock while waiting, intentionally serializing request starts for that host.
The HTTP Semantics specification defines the status and retry-related header behavior the client must interpret.
How Do You Process Thousands of URLs Without Creating Thousands of Tasks at Once?
Method 2: Use a Bounded Work Queue
Best for: Large URL lists or continuously discovered pages.
Step 1. Put approved URLs into a durable queue or database with a unique key, attempt count, and next eligible fetch time. An in-memory asyncio.Queue is useful within a process but cannot recover a machine crash.
Step 2. Start a fixed number of workers that each pull one URL, call the bounded fetcher, validate the record, and acknowledge the task only after writing durable output.
Step 3. Record failures with a reason and retry time. Dead-letter URLs that exhaust retries or violate policy. Apply backpressure when the parser or storage tier falls behind.
Limit:asyncio.gather over a million URL coroutines consumes memory even with a semaphore. The queue bounds task creation and supplies a place to checkpoint work.
Step 1. Get an Nstdata API key and define the approved URL scope. The Crawl API documents x-api-key authentication and site jobs with maxDepth, maxPages, includeUrls, and excludeUrls.
Step 2. Submit a bounded job to Nstdata Crawl. Poll status and retrieve completed pages through the documented endpoints. Nstdata handles page discovery, recursive crawl, JavaScript rendering, and proxy routing.
Step 3. Parse the returned Markdown, HTML, raw data, links, screenshots, or PDF according to your application need. Validate content and store provenance before indexing or training.
Limit: The managed API does not replace your source permissions, scheduler across sites, or arbitrary product-schema extraction. Do not use it to evade access restrictions.
Why Do Concurrent Requests Fail or Return Bad Data?
429 means the source is rate limiting; reduce starts per host and honor the response's retry guidance. A timeout may indicate network trouble or a slow source; retry a small number of times and record failure instead of retrying indefinitely. A 200 with an empty title can be a JavaScript shell, consent page, or parser mismatch. Check the returned HTML and browser network behavior before changing concurrency.
Monitor accepted records per second, median and tail latency, 429 rate, retry count, and queue age by host. A high request count with low accepted-record yield is a quality problem. If one domain starts failing, pause that domain while other approved sources continue.
How Do You Make the Example Production-Safe?
Move URL input to a durable queue and store one terminal outcome per URL. A large gather call creates a coroutine for every input before the semaphore runs, so it is unsuitable for a very large manifest. A fixed worker pool reading from a queue keeps memory bounded and allows a crashed worker's lease to expire. Acknowledgement should follow durable storage of the accepted record or failure outcome.
Add a source-specific policy record. The example's one-second interval and three per-host connections are illustrative. Real settings should come from the site's allowed access pattern and a pilot. Different hosts may need different start intervals. A global semaphore protects your process, but it does not prevent several distributed workers from collectively exceeding a host budget. For distributed crawls, put rate state in a shared scheduler or assign each host to one coordinated worker group.
Improve retry handling before large runs. Numeric Retry-After values are supported in the example; HTTP-date values need parsing. Add a jittered backoff to avoid many workers retrying at the same instant. Set a maximum elapsed time and attempts, then dead-letter tasks that cannot succeed. Do not retry 401, 403, or malformed URLs as though more attempts would grant access.
Finally, keep HTML parsing off the event loop if it becomes CPU-heavy. Small Beautiful Soup operations may be acceptable, but parsing very large pages or running complex extraction in the same loop can delay all requests. Measure event-loop lag and move heavy parsing to a worker process or a separate stage when it matters. Async network I/O and parallel CPU work solve different bottlenecks.
What Should You Monitor in a Concurrent Scraper?
Record latency and outcomes by host. A global average hides the fact that one source is timing out or returning soft error pages. Track in-flight requests, request starts per host, 429 rate, timeouts, response bytes, accepted-record yield, retry attempts, and queue age. Alert on sustained quality degradation, not merely process crashes.
Use an acceptance sample after changing concurrency. Higher throughput can increase incomplete responses, challenge pages, or source load. Compare accepted records per second and per unit cost, and verify that source-side rules are still respected. If accepted yield falls while request count rises, reduce the rate and inspect a sample before scaling further.
Keep raw and accepted counts separate in the dashboard. A burst of 200 responses is not success if most pages contain an error template. Validate an expected page marker, title, content type, and required fields before counting a record as accepted. This gives the operations team a signal to pause one source while the rest of the queue continues.
No. A semaphore limits simultaneous work. Use a per-host pacing mechanism to control request starts over time.
Q: Can aiohttp scrape a JavaScript-rendered page?
aiohttp fetches HTTP responses but does not execute page JavaScript. Use a browser-rendering or managed Crawl path for content generated after load.
Q: Should I use asyncio.gather for one million URLs?
No. Feed a bounded set of workers from a durable queue so memory use and job recovery remain controlled.
This Python tutorial moves from one static page to validated records and managed rendering. It includes runnable examples, failure diagnosis, source controls, and an AI-ready data contract.
Marcus Chen
Oct. 29th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
