TL;DR
- Moving from 1,000 to 100 million pages is a change in scheduling, storage, quality control, and source governance, not just a larger concurrency setting.
- Keep a durable URL frontier with canonicalization, per-host budgets, retry state, and checkpoints; a process-local list cannot recover a long crawl.
- Separate page acquisition from parsing, validation, and indexing so a slow downstream stage applies backpressure instead of losing data.
- Measure accepted records per dollar and per hour, plus freshness and error rates by domain; raw request count hides poor data quality.
- Nstdata Crawl can handle approved site discovery, recursive crawling, JavaScript rendering, and proxy routing. A 100-million-page program still needs an external scheduler, storage, governance, and capacity planning.
What Changes When Web Scraping Grows from 1K to 100M Pages?
The main change is that failures become normal operating events. At 1,000 pages, a developer can retry a script and inspect its output. At 100 million pages, hosts change markup, URLs duplicate, workers restart, some jobs remain incomplete, and records must be corrected or deleted later. The system has to make every page's state observable and recoverable. The Nstdata Crawl product is a managed option for the acquisition and conversion steps discussed here.
Treat 100 million as a planning scale, not a claimed quota of any single API or job. Even at a sustained 100 accepted pages per second, 100 million accepted pages take about 11.6 days before re-crawls, pauses, and rejected pages. The arithmetic is pages / accepted_pages_per_second / 86,400. Capacity must be modeled against the actual site mix and the provider's current limits.
The automated collection guide covers the transition from scripts to recurring jobs.
What Should You Build Before Increasing Concurrency?
Build a durable frontier and a stable record contract. The frontier stores each normalized URL, source, discovery parent, priority, earliest eligible fetch time, attempt count, status, and last error. A unique key should prevent the same URL from being enqueued repeatedly. Normalize only what you can defend: removing a fragment is usually safe for ordinary HTML navigation, while collapsing query parameters can erase distinct product or search pages.
Define the output record before you scale the fetch layer. A minimal accepted page record has source_url, canonical_url, fetched_at, status_code, content_hash, parser_version, content, and quality_status. Store raw acquisition artifacts separately from derived text so parser fixes do not require fetching every source again. Preserve a link from each derived record to the raw artifact and the policy decision for that source.
The Robots Exclusion Protocol belongs in the source-specific scheduler, alongside contractual and rate constraints.
How Do You Scale a Crawl in Stages?
Use four stages, each with a measurable exit criterion.
Method 1: Validate a 1K-Page Pilot
Best for: Proving extraction quality and source permissions before adding infrastructure.
Step 1. Select a stratified sample across domains, templates, languages, and JavaScript behavior. Confirm that each source is approved for the intended use.
Step 2. Fetch the sample with a bounded concurrency setting and a timeout. Save raw responses or rendered artifacts with a source URL and timestamp.
Step 3. Review accepted output against source pages. Record duplicate rate, missing fields, blocked or error pages, bytes per page, and accepted pages per second.
Decision gate: If a significant template fails, repair the parser or choose a rendering path before multiplying volume. A higher fetch rate makes a bad parser more expensive.
Method 2: Add a Durable Frontier for 100K to 1M Pages
Best for: Multiple workers, restarts, and selective re-crawls.
Step 1. Put pending URLs in a database or queue with a unique canonical key, lease expiration, next_fetch_at, and attempt count.
Step 2. Give each worker a bounded batch. Mark a URL complete only after its artifact and extraction result are durable. Expired leases should return to the queue.
Step 3. Partition scheduling by host or source and enforce a per-host request budget. Honor applicable crawl policies and slow down on 429, 503, timeouts, and rising error rates.
Decision gate: Restart workers deliberately. The number of lost or duplicated accepted records should be measurable and within your tolerance.
Method 3: Separate Acquisition and Transformation at 10M Pages
Best for: Mixed static and rendered sites with several output consumers.
Step 1. Send fetch tasks to an acquisition tier. Put raw HTML or rendered output in object storage with immutable identifiers and checksums.
Step 2. Run extraction, normalization, policy filtering, and quality validation as separate consumers. Version the parser and preserve rejection reasons.
Step 3. Apply backpressure from storage and extraction queues. Pause discovery or reduce request rate when the oldest queued item or storage lag exceeds the service target.
Decision gate: A parser deployment should reprocess stored artifacts without refetching the entire corpus.
Method 4: Govern a 100M-Page Program
Best for: Continuous, multi-source collection with a team owning operations.
Step 1. Allocate a source-specific crawl budget and freshness target. Prioritize high-value changed pages over a full recrawl of every URL.
Step 2. Monitor accepted-page throughput, 95th-percentile age, retry rate, source-level error rate, duplicate rate, storage growth, and unit cost.
Step 3. Reconcile source inventories, deletion requests, job state, and downstream indexes. Sample quality after each parser or site-template change.
Decision gate: The program can explain where each record came from, when it was collected, whether it is current, and how to remove it.
The browser rendering at scale guide discusses the distinct capacity limits of rendered pages.
Where Does Nstdata Crawl Fit in This Architecture?
Nstdata Crawl is the managed acquisition layer for selected sites. The Crawl API accepts site jobs with maxDepth, maxPages, includeUrls, and excludeUrls, and supports page discovery and recursive crawling. It can render JavaScript and return Markdown, HTML, raw data, links, screenshots, or PDF. Built-in proxy routing removes the need to configure separate proxy Host, Port, Username, Password, and Protocol fields for a standard managed call.
The application still owns the frontier across sites, queue leases, quality rules, object storage, cost reporting, and deletion propagation. Nstdata's documented job endpoints can be polled for status and pages. Do not interpret maxPages as a promise that a single job can process a whole 100-million-page corpus; validate limits and throughput in your account before setting a production plan.
The Playwright Python documentation is a starting point when a team elects to operate its own browser tier.
How Do You Prevent High-Volume Crawls from Failing Silently?
Track states, not just HTTP outcomes. A 200 page can contain a login wall, empty shell, cookie banner, or a repeated template. Validate page title, content length, required fields, language, and expected domain. Store rejected_reason values such as access_page, empty_render, duplicate, schema_mismatch, and policy_excluded. Review a sample of both accepted and rejected pages.
Use idempotent writes keyed by canonical URL plus content or fetch version. Make retries safe after a worker crash. Keep an error budget per domain; a global success rate can hide one broken high-value source. When a host is unhealthy, pause that source and preserve other sources' progress. The Scrapy architecture documentation shows why scheduling, downloading, and item processing are separate responsibilities.
The website page-discovery guide helps when the URL frontier begins with incomplete coverage.
How Do You Estimate Capacity Without a Misleading Requests-Per-Second Target?
Model each source class separately. Start with approved URLs, average bytes per raw and derived page, fraction requiring rendering, expected acceptance yield, revisit frequency, and permitted request budget. A site with small static pages may be network-bound; a rendered site may be browser- or cost-bound; a table extraction workload may be parser-bound. One average across all domains conceals the bottleneck.
Estimate three volumes: acquisitions, accepted records, and stored bytes. If 100 million accepted records require a 70% acceptance yield, the pipeline must acquire roughly 143 million candidates, before re-crawls. That is illustrative arithmetic, not a forecast. Use measured pilot yield and a confidence range. Multiply storage estimates by raw artifacts, derived versions, and retention periods; versioning can make storage grow faster than URL count.
Set a daily budget per domain and a global budget for spending and downstream queue age. If the extractor cannot consume new artifacts as quickly as the fetcher produces them, increase extraction capacity or slow acquisition. Adding fetch workers in that state only enlarges the backlog and pushes freshness further behind.
How Do You Design Re-crawls Without Starting Over?
Give each accepted URL a last_fetched_at, content hash, change score, and next_fetch_at. Revisit frequently changing pages sooner and stable pages later, within source policy. New links discovered from a changed page can enter the frontier with higher priority than unchanged older pages. This supports continuous coverage without a full reset of the 100-million-URL list.
Distinguish a missing page from a transient failure. A 404 may mean permanent removal, a moved URL, or a temporary source error. Record the response and recheck according to policy before deleting a valued record. If a page changes canonical URL, preserve the old identity and redirect relationship so downstream indexes can be updated and duplicate documents removed.
Re-crawling must also cover parser changes. When raw artifacts are retained and permissions allow it, a new parser version can reprocess old captures without a network request. That is particularly useful for fixing a field-mapping bug across a large corpus. It does not make the content current; a separate fetch is required for freshness.
FAQ
Q: Can I scale scraping by setting concurrency to 1,000?
No. Concurrency does not provide durable job state, per-host pacing, storage, or data-quality checks. Increase it only after a pilot establishes safe source-specific budgets.
Q: Does Nstdata Crawl replace my data pipeline?
Nstdata Crawl handles acquisition and conversion. Your application still needs scheduling across sources, validation, storage, indexing, and governance.
Q: What is the most useful throughput metric?
Accepted, policy-approved records per unit of time and cost are more useful than completed HTTP requests.
