Retrying and Error Handling in Production Scrapers
TL;DR
A production scraper needs terminal outcomes for every URL: accepted, rejected, deferred, or failed with a reason. An HTTP 200 alone is not success.
Retry transient network faults and selected statuses such as 408, 429, and 5xx; do not automatically retry 401, 403, or a parser mismatch.
Honor Retry-After, use capped backoff with jitter, and keep a source-specific request budget so retries do not amplify overload.
Make writes idempotent and store attempt count, next eligible time, and last error; process restarts should not lose or duplicate accepted records.
Nstdata Crawl provides managed task state and recovery for acquisition, while your application still validates page meaning and controls downstream retries.
Why Do Scrapers Need More Than a Retry Loop?
A retry loop addresses one class of failure: a request that may succeed later. A production scraper also needs to know whether it reached the intended page, whether extraction produced a valid record, and whether the result was stored. A 200 response can be a consent screen, login prompt, empty JavaScript shell, or fallback homepage. Retrying that same response indefinitely wastes requests and may overload the source.
Nstdata Crawl can handle browser rendering and acquisition, but the application still needs its own accepted-record and retry policy.
Separate four boundaries: transport, HTTP response, content, and persistence. Transport errors include DNS, connection, and timeout failures. HTTP status describes the server response but not its semantic content. Content validation decides whether the page is usable. Persistence confirms the accepted record is durable. Give each boundary a distinct error code and owner. can simplify the acquisition boundary, but it does not know whether a extracted price matches your product variant.
Retry only when the failure is plausibly temporary and the source permits another request. The current Scrapy RetryMiddleware documentation includes 408, 429, 500, 502, 503, 504, 522, and 524 in its default retry statuses. That is a useful reference, not a rule to copy blindly: source-specific behavior and Retry-After matter.
Symptom
Likely decision
Reason
Connection reset or timeout
Bounded retry
May be transient; log the cause
429
Defer using Retry-After and reduce rate
Source is limiting requests
503
Back off; retry within budget
Temporary unavailability is plausible
401 or 403
Stop and review access
Repeated attempts do not grant permission
404
Verify URL or mark missing
Usually not fixed by rapid retries
200 with consent/login page
Reject and investigate
Transport worked; content is wrong
Missing field
Parser or source review
More identical requests may not repair markup
Never rotate identities, change headers, or discard source restrictions merely to force a denied request through. Review the approved access path or use an official API. The HTTP Semantics specification is the authority for status code and header meaning.
What Do You Need Before Implementing Retries?
Define a per-source request policy, a maximum attempts and elapsed-time budget, timeout values, accepted-page markers, and durable task storage. Install a client such as requests if using Python. Record source_id, URL, attempt number, last status/error, next_fetch_at, and a stable record key. If the application sends multiple workers, coordinate per-host pacing across them; a local sleep in each worker does not limit their combined rate.
Keep credentials in a secret manager or environment variable. For managed Crawl calls, obtain an Nstdata API key from Dashboard → Crawl Playground and send it as x-api-key. Built-in proxy routing means the normal Crawl request does not need separate proxy connection fields.
How Do You Implement Bounded Retries in Python?
Method 1: Retry Temporary HTTP and Network Failures
Step 1. Use one session with explicit connect and read timeouts. Identify the client and operate only within the source's allowed access pattern.
Step 2. Handle retryable statuses, Retry-After, and network exceptions separately:
Step 3. Validate the returned page before accepting it. Check content type, canonical source, title, required fields, and known soft-error patterns. Record an explicit rejected_reason if a 200 response is not the intended page.
This example is deliberately bounded. In production, put maximum elapsed time around retries and validate any very large Retry-After against the job budget by deferring the task rather than holding a worker asleep. The code does not retry 401 or 403; raise_for_status() surfaces them for policy review.
How Do You Keep Retries Safe After a Worker Restart?
Method 2: Use a Durable Task State Machine
Step 1. Store URL tasks in a queue or table with a unique key and states such as pending, leased, retry_pending, accepted, rejected, and failed_terminal.
Step 2. Give a worker a time-limited lease. Write the raw artifact and validated result before acknowledging the task. If the worker crashes after writing but before acknowledgement, the next worker must recognize the same record version and avoid a duplicate insertion.
Step 3. On a retryable failure, save attempt count and next_fetch_at. On a policy denial or content rejection, record the terminal reason and stop automatic requests. Reconcile submitted and terminal counts at the end of each run.
A content hash is useful for change detection but should not be the only identity. Two different URLs may legitimately share text, and one URL may produce many versions. Use source ID and canonical URL for entity identity, plus hash and fetch time for versions.
How Do You Diagnose Silent Scraper Failures?
Method 3: Add Semantic Acceptance Checks
Step 1. Save a small fixture set: normal page, redirect, missing field, known consent page, error page, and JavaScript shell.
Step 2. Define required markers for each page type. A product record might require product ID, currency, and a visible price; an article needs a title and body. Validate those markers after retrieval and before storage.
Step 3. Sample both accepted and rejected records after source or parser changes. Alert on acceptance yield and rejection reason by domain, not just request error rate.
How Can Nstdata Help Keep Acquisition Errors Observable?
Nstdata Crawl is a Web Crawling API for LLM-ready data that accepts a reviewed page or bounded site job and exposes task status and results. Its documentation distinguishes an outer HTTP response from nested task success, status, and error codes; an application must inspect both. Built-in retries and proxy orchestration can reduce infrastructure work, but a task may still fail, return a denied page, or produce content that fails your schema.
LLM-ready output: Markdown and cleaned HTML let downstream RAG or extraction code focus on relevant content; raw data helps diagnose a bad conversion. Links, screenshots, and PDF provide navigation and visual evidence. Choose only the formats a consumer needs, and inspect markdownRef, htmlRef, or other references when large outputs are stored separately.
Built-in proxy routing: The standard API request takes x-api-key and page options without separate Host, Port, Username, Password, or Protocol proxy fields. This reduces setup and credential handling for permitted collection. Proxy routing does not authorize access denied by a source.
Task visibility: Site jobs report completed, failed, and pending counts through the documented status route. Your app should still reconcile accepted and rejected business records.
How Do You Add Nstdata Crawl to the Retry Workflow?
Step 1. Obtain an API key in Dashboard → Crawl Playground and select an approved test URL. Set a request timeout within current documented limits.
Step 2. Submit a single page and save the raw response for diagnostic comparison:
curl--request POST 'https://api.nstdata.io/api/v1/crawl/scrape/submit-sync'\--header"x-api-key: ${NSTDATA_API_TOKEN}"\--header'Content-Type: application/json'\--data'{"url":"https://example.com/","formats":["markdown","html"],"timeout":60000}'
Step 3. Check the outer HTTP status, then data.success, data.status, any task code, and data.data.metadata.statusCode. Fetch a reference token through GET /api/v1/crawl/storage/read?st={ref} when needed. Apply your semantic acceptance rules and store a terminal outcome. The snippet follows official documentation but cannot be live-executed without an account key.
In production, cap retries at both the provider and application layers so the same failing page does not repeatedly consume budget. For permitted browser interactions beyond page conversion, Nstdata Cloud Browser is a distinct product. The next action is a failure drill: make one page time out, one return a soft error, and one pass, then confirm every task ends in the correct state.
When Should a Scraper Open a Circuit Breaker?
Pause a source when repeated failures indicate that more requests are unlikely to help. A sudden rise in 403 responses, challenge pages, empty renders, or parser rejections should open a source-specific circuit and alert the owner. Use a short, controlled probe only after the access or source issue has been reviewed. A circuit breaker protects the target from unnecessary load and prevents a bad template from contaminating the dataset. Record when it opened, why, and who authorized resumption.
