A reliable scraper diagnoses failures by layer: request construction, network transport, HTTP response, rendered page, extraction, and storage.
Retry only temporary failures within a bounded budget; a 401, 403, broken selector, or disallowed path needs review rather than another attempt.
A 200 response can still contain a consent screen, login prompt, empty JavaScript shell, or unrelated fallback page, so validate content before accepting data. For a browser-dependent page, the JavaScript scraping guide explains how to distinguish static HTML from rendered content.
Store each URL's attempt count, last error, next eligible time, and terminal outcome so worker restarts do not duplicate records.
Nstdata Crawl is a Web Crawling API for LLM-ready data that can handle page retrieval, JavaScript rendering, proxy routing, retries, and task state; your application still owns semantic validation and downstream writes.
How do you solve common web scraping errors?
Solve a web scraping error by identifying which part of the collection pipeline failed, then apply a fix to that layer. Repeating the same request without classifying the failure wastes capacity and can put unnecessary load on the source.
Start with a trace for one affected URL. Record its canonical URL, fetch time, HTTP status, response content type, elapsed time, redirect destination, parser version, retry count, and validation result. Do not save cookies, authorization headers, API keys, or private response bodies in ordinary logs.
The first investigation should answer four questions: did the request reach the intended host; did the response represent the requested page; did the parser find the expected fields; and did the accepted record reach durable storage? These checks separate transport faults from page changes and application bugs.
For example, a timeout is a transport issue; a 403 is a denial that needs an approved access path; a 200 with no expected title is a content or rendering problem; and a valid record missing from the warehouse is a persistence problem. Treating all four as “scraping failed” hides the repair.
Nstdata already covers several adjacent infrastructure tasks in its . The current can retrieve pages and bounded sites, render JavaScript, route through managed proxy infrastructure, and return multiple representations. It does not decide whether a value satisfies your business schema.
Nstdata Crawl is a Web Crawling API for LLM-ready data that can take page acquisition and conversion out of an application's retry loop. It addresses the infrastructure side of errors such as dynamic content, task tracking, and proxy routing while leaving source policy and business validation with your team. Its Markdown, HTML, JSON, links, screenshot, and PDF outputs let a pipeline compare readable text with structural or visual evidence. It fits teams that need repeatable page or bounded site jobs but do not want to maintain every browser worker and collection queue. It is not a substitute for permission checks or schema validation.
Rendered retrieval: Use JavaScript rendering for approved pages whose content appears after browser execution; do not send every static page through a browser.
Inspectable outputs: Compare Markdown, HTML, and visual artifacts to diagnose whether an error came from retrieval or extraction.
Task state and managed routing: Use task IDs and documented status fields for acquisition visibility; standard Crawl requests do not require separate proxy connection fields when managed routing is selected.
What should you prepare before troubleshooting a scraper?
Prepare an authorized target list, one reproducible failing URL, a known-good page, access rules, a request budget, and an expected record schema before changing code. Without these items, it is difficult to tell whether a fix repaired the scraper or simply changed the request.
For a Python workflow, record the Python and package versions, the configured timeout, user agent, concurrency, and any proxy settings. Keep keys in environment variables or a secret manager. For a managed crawl API, obtain a Nstdata API token through the dashboard and check the current Crawl documentation for the endpoint and output options before running a request.
Define an acceptance rule for each page type. A job page might need a title, employer, location, and a stable source URL. A product page might need a product identifier, currency, and price. Set a policy for missing or malformed fields: reject, quarantine, or accept with an explicit null. Do not silently invent values.
Why can a scraper fail even when the request returns 200?
A scraper can receive HTTP 200 and still fail because the response body may not be the intended content. Common causes include an anti-bot notice, consent page, authentication wall, JavaScript placeholder, geolocation variation, soft 404, or a redesign that moved the target element.
Check the final URL after redirects, the page title, canonical link, expected text markers, and required extracted fields. Compare a small sample of the raw or rendered source against a known-good fixture. If the page is a denial or asks for authorization, stop and review the approved access path; do not switch identities to force access.
Diagnose the Failure Before Retrying
Use Nstdata Crawl to inspect task status and cleaned page outputs in a bounded web collection workflow.
Retry only when the error is plausibly temporary and the source permits another request. The HTTP Semantics specification defines status behavior, while RFC 6585 defines 429 Too Many Requests. A 429 should trigger a lower request rate and respect for Retry-After, not a faster retry.
Symptom
Likely cause
Next action
DNS, connection reset, or timeout
Temporary network fault or a bad route
Retry within a short attempt and elapsed-time budget
429 Too Many Requests
The source is limiting requests
Defer, honor Retry-After, and reduce shared concurrency
5xx response
Temporary source or intermediary failure
Back off and retry only within the source policy
401 or 403
Missing authorization or access denied
Stop automatic retries and review permission
404
URL changed or resource was removed
Verify the URL, then record a missing item
200 with empty or unrelated page
Rendering, redirect, consent, or soft error
Inspect the page and validate before storage
Missing field after a redesign
Parser or source schema changed
Update and test the parser against fixtures
The Requests maintainers document request exceptions and response handling in the Requests Quickstart. Scrapy's retry middleware similarly distinguishes retryable failures from failures that exceed its configured limits; see the Scrapy downloader middleware reference.
How do you troubleshoot common scraper errors?
Use a repeatable sequence: capture the actual response, classify the failing layer, make one change, and verify the result against a known acceptance rule. Change one variable at a time so the repair remains explainable.
Method 1: Diagnose network and HTTP errors
Use this method when requests time out, return error statuses, or behave differently across runs.
Step 1: Reproduce one URL with explicit timeouts
Run a single request with separate connection and read timeouts. Capture the final URL, response status, content type, and elapsed time. Avoid testing a large batch while the root cause is unknown.
Step 2: Classify the status before retrying
Use bounded retries for temporary network faults and selected 5xx responses. For 429, record Retry-After and pause the affected source. For 401 or 403, stop and confirm that the access route is authorized. A retry loop must not turn a denial into a high-volume request stream.
Step 3: Compare the response with an accepted sample
Check page identity and required markers. If the response is a challenge page, login prompt, or empty shell, mark it rejected with a reason instead of passing it to the parser.
Method 2: Find parser and selector drift
Use this method when requests succeed but extracted records are empty, incomplete, or malformed.
Step 1: Save a small fixture set
Keep representative pages for the current layout, a page without optional fields, and a known error page. Remove credentials and personal data before storing fixtures.
Step 2: Check stable page identifiers
Prefer documented API fields, semantic labels, stable attributes, or structured data over deeply nested generated CSS selectors. Validate expected field count and type after extraction.
Step 3: Reject unexplained changes
Quarantine records that break the schema. Review a diff between the accepted fixture and the latest response, then update the parser and its tests together.
Method 3: Make retries safe across worker restarts
Use this method when jobs run asynchronously or a restart can repeat already-fetched URLs.
Step 1: Give every task a stable identity
Use a source identifier and canonical URL as the task key. Keep the fetch timestamp and content hash separately so a changed page is a new version, not an accidental duplicate.
Step 2: Persist state before acknowledging work
Save the artifact and accepted record before marking a task complete. If a worker stops between those operations, the next worker should detect the existing result rather than insert a duplicate.
Step 3: Write a terminal reason
Store states such as pending, retry pending, accepted, rejected, and failed terminal. Save the attempt number and next eligible time for deferred work. At the end of a run, reconcile submitted, accepted, rejected, and failed counts.
Method 4: Move page acquisition to Nstdata Crawl
Use Nstdata Crawl when your team needs to retrieve permitted public pages but does not want to operate its own browser workers, proxy routing, task queue, and content conversion for each application.
Step 1: Create an API token and choose a bounded test URL
Open Dashboard → Crawl Playground, create or retrieve an API token, and choose one public URL that your team is allowed to collect. Keep the token in an environment variable.
Step 2: Submit a page request with only the required outputs
The current API documentation shows a synchronous page endpoint and an API key header. A request can ask for Markdown and HTML for comparison:
curl--request POST 'https://api.nstdata.io/api/v1/crawl/scrape/submit-sync'\--header"x-api-key: $NSTDATA_API_TOKEN"\--header'Content-Type: application/json'\--data'{"url":"https://example.com/","formats":["markdown","html"],"onlyMainContent":true}'
This follows the current documented request shape but is not a live account test; it requires a valid Nstdata API token. The standard Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. If a workflow needs a custom proxy source, configure that supported option explicitly.
Step 3: Inspect task state and validate the result
Check both the outer API response and the nested result status, success flag, metadata, and error fields. For large artifacts, retrieve the returned reference through the documented storage endpoint; never invent a reference token. Accept the page only after your application confirms its source identity and required content.
Nstdata returns formats such as Markdown, cleaned HTML, raw page data, links, screenshots, and PDF. Markdown is useful for LLM prompts and RAG ingestion; HTML is better when your own selectors or DOM rules matter; screenshots help diagnose layout changes; links support follow-up discovery. Format conversion does not guarantee that the extracted business values are correct.
The Crawl pricing page describes the current billing model and plan limits. Workload cost depends on URL volume, output choice, proxy routing, retries, and concurrency. Estimate from a bounded sample and include failed or rejected pages in your own cost-per-accepted-record calculation.
How do you keep a scraper error from becoming bad data?
Keep errors visible in a durable ledger and connect them to data-quality checks. A low HTTP error rate can hide a broken parser, while a high request count can conceal repeated duplicates.
Track status counts by source and page type: requested, accepted, rejected, deferred, and terminally failed. Also track empty-content rate, required-field coverage, duplicate rate, and output size. Alert on sudden changes rather than relying only on a generic “job completed” event.
When a managed crawl task reports completion, that confirms a task state, not your dataset's business quality. Your consumer must still check title, canonical URL, required fields, date format, and acceptable value ranges. If one domain changes its structure, isolate its failures so it does not invalidate unrelated sources.
What should you do next after fixing a scraper?
After fixing an error, replay the affected URL and a small representative sample, compare accepted output with the intended schema, and only then resume the bounded job. Record the code or configuration change with the fixture and error category that motivated it.
Use the Python HTML extraction guide if the fetch succeeds but text cleanup remains the problem. If repeated page retrieval, rendering, task tracking, or format conversion is the operational bottleneck, evaluate Nstdata Crawl as the web collection layer and keep validation and storage in your own application.
Q: Should every failed scraping request be retried?
No. Retry transient network faults within a bounded budget, defer 429 responses according to Retry-After, and stop on access denials or parser defects.
Q: Does HTTP 200 mean the extracted record is valid?
No. Confirm that the response is the intended page and that required fields pass your schema checks.
Q: Is a 403 response a temporary scraper error?
Not by default. Treat it as a denial, stop automatic retries, and review the target's published access path and your authorization.
Q: What should be stored for a failed URL?
Store its canonical URL, source, status, attempt count, last error, next eligible time if deferred, and a terminal outcome. Exclude credentials from logs.
Q: Does Nstdata Crawl replace application-level validation?
No. Crawl handles acquisition, rendering, task state, and output conversion; your application determines whether a page satisfies its data contract.
Lena Zhou
Nov. 13th 2026
Experience Nstproxy Crawl - Start Your Free Trial Today
Crawl entire websites with a single API request
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
