TL;DR
- Large-scale scraping fails first at coordination, duplicate work, validation, and retriesβnot raw request speed.
- Use a durable queue, idempotent jobs, per-domain budgets, static-first retrieval, bounded browser pools, and append-only results.
- Partition by policy and target behavior, checkpoint every stage, and calculate cost per accepted page.
- Managed Crawl and proxy operations can remove infrastructure work, but they do not replace data contracts or business validation.
Scaling a scraper to millions of pages means scaling decisions and recovery paths, not launching millions of simultaneous requests.
The Bottlenecks That Appear First
Duplicate URLs inflate work, retries amplify incidents, browser workers exhaust memory, target-specific limits create head-of-line blocking, and incomplete pages pass as successes. Nstdata Crawl can remove parts of the retrieval layer, but the architecture still needs idempotency and validation. The HTTP semantics specification helps classify responses; OpenTelemetry helps connect traces and metrics; the Robots Exclusion Protocol informs crawl policy.
A Production Architecture
discovery β canonicalization β policy admission β durable queue β static workers β browser escalation β extraction β semantic validation β append-only storage β export
