You can submit an entire website crawl with one POST /api/v1/crawl request to Nstdata Crawl. The request starts an asynchronous job; polling and paginated result retrieval are still required afterward.
A safe full-site crawl is bounded, not literally unlimited. Set maxDepth, maxPages, include/exclude URL patterns, query handling, output formats, and a timeout before the crawler follows links.
Depth and page limits solve different problems. Depth limits how many link hops the crawler travels; page count caps total work even when the link graph is wide.
Canonicalization and traps determine data quality. Calendar paths, faceted navigation, tracking parameters, redirects, and duplicate canonicals can consume a crawl long before valuable pages are reached.
Treat completion as a dataset-quality event. Inspect completed, pending, and failed counts; paginate every result page; deduplicate; and verify coverage against a sitemap or known URL set.
An “entire website” is rarely a finite, clean list. One domain can expose millions of parameter combinations, calendar links, search pages, locale duplicates, and JavaScript routes. A useful crawl therefore needs a starting URL plus explicit boundaries.
Nstdata Crawl packages discovery, page retrieval, rendering, extraction, proxy routing, task state, and artifact storage behind one API. This tutorial submits a bounded site job, controls scope, polls status, retrieves every result page, and checks whether the finished dataset is complete enough for its intended use.
Nstdata Crawl starts from one URL, discovers same-site links, and processes eligible pages according to the limits in the request. The submission returns a task identifier rather than holding the HTTP connection open until the whole site is finished.
The phrase “single API request” applies to job submission. A production client must later call status and page-result endpoints. This asynchronous design is appropriate because an entire site can take longer than a normal request timeout and can produce a paginated dataset.
Requested page artifacts, such as Markdown or HTML
maxDepth
Maximum link distance from the seed
maxPages
Maximum number of pages processed
includeUrls
Patterns allowed into the crawl
excludeUrls
Patterns removed from the crawl
ignoreQuery
Reduces query-parameter duplicates when enabled
onlyMainContent
Focuses extraction on primary page content
timeout
Bounds page-processing time
Nstdata’s crawler glossary defines the discovery mechanism, while its guide to scraping versus crawling explains why discovery and extraction are separate operations inside one job.
Why Crawling an Entire Website Is Hard
Full-site crawling is hard because websites expose graphs, not directories. A crawler must decide which discovered URLs represent new content, which are duplicates, and which lead into infinite or low-value spaces.
client-side links that appear only after JavaScript execution;
soft 404 pages that return HTTP 200;
large files and endpoints that are not HTML pages.
Robots rules and sitemap discovery provide important signals. The Robots Exclusion Protocol standardizes robots.txt behavior, and the Sitemap protocol defines the common XML URL-list format. Neither source grants permission to collect data; you must also respect terms, authentication boundaries, copyright, privacy, and applicable law.
Prerequisites
You need an Nstdata account, a Crawl API key, an authorized public seed URL, and a clear definition of required coverage. Decide whether you need page text, links, HTML, screenshots, or another supported artifact before submission.
Create a small acceptance plan:
expected URL families, such as /docs/ or /products/;
excluded families, such as /account/, /cart/, /search/, and calendars;
maximum depth and page budget;
expected language and canonical-host rules;
minimum content checks for accepted pages;
refresh and deletion policy for downstream storage.
Start with a low maxPages value against a representative section. A bounded pilot reveals URL traps without consuming the full job budget.
Turn One URL into a Site Dataset
Use Nstdata Crawl to discover, render, and return pages within explicit limits.
Submit one asynchronous job to the current site-crawl route. A credential-free probe on September 4, 2026 returned HTTP 401, confirming that /api/v1/crawl is live and requires authentication. The request below is documentation-aligned but could not complete without an account key.
The expected response envelope contains a task ID and a processing state. Do not copy an illustrative ID into later requests; always persist the exact identifier returned for your submission.
Request only the formats you will use. Markdown is appropriate for LLM and RAG ingestion, while HTML helps when your parser needs selectors or semantic markup. Multiple formats increase artifact volume and downstream handling.
Nstdata’s Crawl launch overview describes the broader product workflow, including site discovery and LLM-ready outputs.
Control Crawl Depth and Scope
Control scope by combining depth, page count, path patterns, and query normalization. No single setting is sufficient.
Choose maxDepth by information architecture
Depth zero or one is useful for validating the seed and immediate navigation. A documentation hub may need two or three hops to reach topic pages. A high depth does not guarantee coverage if important pages are reachable only through sitemaps or JavaScript search.
Depth also depends on the chosen seed. Starting at the domain homepage can waste hops in marketing pages; starting at /docs/ gives the same depth budget more relevance.
Treat maxPages as a hard budget
Page count prevents a wide link graph from expanding indefinitely. Set a pilot budget below your expected corpus, inspect the discovered URL mix, then raise it only when valuable pages dominate.
If the job reaches maxPages, completion does not mean the entire intended site was covered. It means the bounded job stopped at its configured ceiling.
Use include rules before exclude rules
An allowlist such as *example.com/docs/* is easier to reason about than dozens of exclusions. Add exclusions for known low-value subtrees inside the allowed section.
Test pattern behavior against sample URLs before launch. A misplaced slash or host pattern can silently exclude every page or admit unrelated subdomains.
Normalize query parameters carefully
Enable ignoreQuery when parameters do not change meaningful content, such as tracking and sort values. Do not discard query strings when they select a real locale, product variant, documentation version, or pagination state that your dataset requires.
URL comparison must follow consistent parsing rules. the WHATWG URL Standard documents modern URL parsing behavior; avoid ad hoc string splitting for hosts, paths, and queries.
Poll the Crawl Status
Query the job state with the returned task ID. Replace the placeholder below with your real identifier:
curl--request GET \--url'https://api.nstdata.io/api/v1/crawl/YOUR_TASK_ID'\--header'x-api-key: YOUR_NSTDATA_API_KEY'
Use bounded exponential backoff with jitter rather than polling continuously. Stop on a documented terminal state and enforce an overall deadline in your application.
Inspect the response body, not only HTTP 200. Nstdata’s current model can report task-level failure information inside a successful HTTP envelope. Record total, completed, pending, and failed counts when present, along with non-secret request and task identifiers.
Do not automatically resubmit a whole site because a few pages fail. Retrieve page-level outcomes, classify failures, and retry only eligible URLs. Authentication failures, disallowed targets, parse errors, and transient timeouts need different responses.
Retrieve Every Crawled Page
Fetch the first result page after the crawl reaches a usable state:
curl--request GET \--url'https://api.nstdata.io/api/v1/crawl/YOUR_TASK_ID/pages?limit=50'\--header'x-api-key: YOUR_NSTDATA_API_KEY'
If the response contains nextCursor, request the next page and continue until no cursor remains. Stopping after the first API response is a common reason a completed crawl appears to contain only part of the site.
Large artifacts may arrive as reference tokens such as markdownRef or htmlRef. Resolve references through the documented storage endpoint; never construct or modify storage tokens yourself.
Persist at least the requested URL, final URL, canonical URL when available, status, title, language, content hash, crawl time, and output reference. Keep page identity separate from pagination cursors, which are transport state rather than document IDs.
Verify Coverage and Data Quality
A crawl is successful when it covers the required corpus with acceptable page content, not merely when its status says completed. Compare the result set against the site’s sitemap, a known navigation tree, or a manually labelled sample.
Calculate:
expected URLs discovered;
discovered URLs processed;
pages with meaningful content;
unique canonical pages;
duplicates and redirects;
failures by reason;
URL budget consumed by each path family.
Inspect a stratified sample from shallow and deep paths. Check JavaScript-dependent pages, tables, code blocks, pagination, locale variants, and known soft 404s. If the crawl budget is dominated by low-value paths, tighten include rules before increasing maxPages.
For build-versus-buy context, Nstdata’s open-source web crawler comparison covers crawler frameworks that offer more low-level control but require you to operate scheduling, rendering, storage, proxies, and monitoring.
Handle Failures and Refreshes
Separate submission errors, task errors, and page-quality failures. An invalid request should fail before job creation. A valid job can still contain target timeouts, access denials, parser failures, or empty content. A technically successful page can still be rejected because it is a duplicate, wrong locale, or consent screen.
For recurring crawls, retain content hashes and compare canonical URLs. Reprocess changed pages, add new pages, and remove deleted pages from downstream indexes. Do not append every run indefinitely.
Use a slower refresh cadence for stable archives and a faster one for changelogs, inventory, or policies. Respect caching headers and target capacity where appropriate. If a source provides change feeds or modification timestamps, use them to reduce unnecessary fetching.
Responsible Full-Site Crawling
Full-site access must be authorized and proportionate. Do not use the API to bypass authentication, paywalls, permissions, or technical safeguards. Avoid private pages and minimize personal or regulated data.
When accepting user-supplied seeds, prevent server-side request forgery. Block localhost, private and link-local networks, cloud metadata endpoints, unsafe schemes, suspicious ports, and redirects outside the approved scope. The OWASP SSRF guidance provides a practical threat model.
Keep API keys in approved secret storage, never logs or source control. Task IDs and storage references can provide access to results, so avoid exposing them to unauthorized users.
Full-Site Crawl Checklist
Before submission, verify seed URL, allowed host, include paths, exclusions, query policy, maximum depth, page budget, output formats, and legal authorization. During the job, monitor status with bounded polling. After completion, paginate all results, resolve required artifacts, deduplicate canonical content, review failures, and compare coverage with a known source.
Nstdata Crawl pricing should be reviewed alongside the page budget and proxy traffic for the chosen workflow. Start with a bounded pilot and measure cost per accepted unique page.
One Submission, Explicit Boundaries, Verified Results
One API request can start a complete site workflow, but good crawling still depends on deliberate boundaries and verification. maxDepth shapes link traversal, maxPages protects the budget, URL patterns focus discovery, and paginated result retrieval turns the job into a usable dataset.
Use the smallest representative crawl first. Once URL distribution, page quality, failure handling, and coverage checks are correct, raise the limits gradually.
Q: Can I crawl an entire website with one API request?
Yes. One POST request can submit a bounded Nstdata site crawl, but the asynchronous workflow still requires later status polling and paginated result retrieval.
Q: What is the difference between maxDepth and maxPages?
maxDepth limits the number of link hops from the seed URL, while maxPages caps the total number of pages the job processes regardless of graph width.
Q: How do I prevent a web crawler from following infinite URLs?
Combine a strict page limit with allowed path patterns, known exclusions, query normalization, canonical deduplication, and a low-budget pilot crawl. Calendar and faceted-navigation paths deserve explicit tests.
Q: How do I know whether the entire website was crawled?
Compare unique accepted results with sitemaps or a known URL inventory, inspect failures and path coverage, and ensure every paginated result page was retrieved.
Compare Firecrawl vs Nstdata Crawl on features, pricing, JavaScript rendering, scalability, and migration costs. Learn when and how to switch.
Lena Zhou
Oct. 16th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
