A web crawler discovers URLs and schedules their retrieval; an extractor turns retrieved documents into useful fields or text.
Crawling starts from seeds and grows through links or sitemap entries within an explicit scope.
URL identity, duplicate prevention, host pacing, and stopping rules determine whether a crawler stays manageable.
JavaScript rendering is needed only when required content is missing from the initial response.
Crawl completion describes a job state, not necessarily complete coverage or an accepted dataset.
Nstdata Crawl is a Web Crawling API for LLM-ready data that can manage collection while your application defines relevance and acceptance.
What Are Web Crawlers?
Web crawlers are programs that discover URLs, place them in a queue, and retrieve documents according to configured rules. A crawler can collect a permitted documentation section for a support assistant, inventory a company's public pages, or observe changes in a research corpus. The same general mechanism supports very different products because each product chooses a different scope and use for the retrieved content.
A crawler does not have to scan the entire internet. A bounded crawler can remain inside one host, follow only approved paths, and stop after a small page count. Those restrictions are often the difference between an intentional dataset and an expensive collection of irrelevant pages.
Nstdata Crawl provides a managed form of this collection layer for developers who need LLM-ready outputs. Your application still decides which content belongs in the resulting dataset.
How Do Web Crawlers Work From a Seed URL?
Web crawlers work by repeatedly taking an eligible URL from a frontier, fetching it, extracting permitted links, and adding new eligible URLs back to the frontier. The frontier is the set of URLs awaiting work. A visited or scheduled-URL registry prevents the same URL from being added repeatedly.
A practical lifecycle is: validate the seed, apply access and scope rules, schedule a request, retrieve or render the page, validate the result, extract links and content, and checkpoint progress. Failure handling belongs in this lifecycle because a temporary outage should not erase the crawler's memory of what it already completed.
Experience Nstproxy Crawl - Start Your Free Trial Today
The Robots Exclusion Protocol defines how crawlers interpret published robots rules. Robots instructions are an important operational input, but they are not authentication or a blanket content license. Applicable rights and access restrictions remain separate questions.
How Is Crawling Different From Scraping, Rendering, and Indexing?
Crawling discovers and retrieves pages, scraping extracts information, rendering executes a page's presentation logic, and indexing organizes accepted information for retrieval. A product can combine these stages, but the distinction helps you diagnose failures.
Stage
Input
Typical output
Common failure
Discovery
Seed pages or sitemap
Candidate URLs
Important pages are unreachable
Retrieval
Approved URL
Response body and status
Timeout or denied access
Rendering
HTML and scripts
Browser-computed document
Required content never appears
Extraction
Retrieved document
Fields, text, or links
Wrong field or missing content
Indexing
Accepted records
Searchable representation
Duplicates or obsolete versions
A parser can be correct while discovery misses most of a site. A browser can render correctly while the model receives an obsolete snapshot. Use the stage names in your logs so the team can identify the actual fault instead of reporting that βthe crawler broke.β
The same separation underlies HTML cleaning for LLM inputs. Cleaner text can improve downstream handling, but collection scope and evidence quality still need their own rules.
What Configuration Does a Crawler Need?
A crawler needs seed URLs, scope rules, a URL-identity policy, resource limits, and an output contract before it can operate predictably. Defaults may be convenient for a demonstration, but a production job should record its configuration with its results.
Control
Question it answers
Allowed hosts
Which sites may the job access?
Included and excluded paths
Which sections belong in the dataset?
Maximum depth
How many link steps may discovery follow?
Maximum pages
How much work may the job schedule?
Query-string policy
Which parameters change content identity?
Rendering policy
Which templates require a browser?
Timeout and retry budget
How long may a request or job consume resources?
Output format
What does the next stage actually need?
Do not remove every query string as a universal deduplication rule. A tracking parameter may be irrelevant, while a language, product, or pagination parameter may identify different content. Similarly, a trailing slash or canonical tag needs a defined interpretation rather than automatic assumptions.
How Do Sitemaps and Links Affect Coverage?
Sitemaps and links affect coverage by supplying different candidate URLs, neither of which proves that every useful page has been found. A sitemap can expose pages not linked from your seed. Link traversal can discover pages absent from the sitemap. Compare both sources with the dataset you expected to collect.
The Sitemaps protocol defines the structure of sitemap documents and indexes. A sitemap entry is a discovery hint, not proof that a URL currently returns valid content or should be included in your application.
For an owned site, compare discovered URLs with the CMS inventory, sitemap set, and known template list. Pages missing from every discovered set may be orphaned or outside the configured boundary. For a third-party site, describe the observed scope without claiming whole-site completeness that you cannot independently establish.
When Does a Crawler Need JavaScript Rendering?
A crawler needs JavaScript rendering when the required information is generated or loaded after the initial HTML response. A page that looks dynamic can still contain the fields you need in its initial document. Inspect the content before paying the operational cost of a browser for every page.
Rendering introduces additional failure states. A document can load while an asynchronous data request fails. A visible placeholder can satisfy a broad selector even though the actual content is absent. A consent screen can replace the expected page. Wait for a meaningful content condition and validate the resulting page type.
If you operate your own browser workers, Playwright's network interface supports observing requests and responses for diagnosis. Browser automation still needs explicit timeouts, resource limits, and an approved target scope.
How Can You Start a Bounded Crawl With Nstdata?
Start a bounded crawl by testing an approved seed in the Playground, setting explicit limits, and validating a sample of returned pages before scaling. A working example should prove that the chosen configuration retrieves the required content rather than merely returning a task identifier.
Method 1: Inspect the collection visually
Step 1: Choose a permitted seed
Select an owned or authorized documentation section with a clear content purpose. Define the pages that should be included and the pages that should never be followed, such as account or internal search paths.
Step 2: Set a small collection boundary
Configure maximum depth, page count, and exclusions in the current Playground. Choose Markdown when your consumer is a text-based AI workflow, and inspect HTML when you need to compare the extraction with the document.
[Screenshot: A bounded Crawl configuration showing the approved seed, page limit, depth limit, rendering setting, and selected formats.]
Step 3: Review accepted pages
Compare the returned pages with a short expected-URL list. Record missing pages, duplicates, failed retrievals, and irrelevant pages. The review gives the next iteration a concrete target rather than an arbitrary instruction to crawl more.
Method 2: Connect a single-page retrieval first
Step 1: Configure the client environment
Install the official nstdata-ai-crawl package and provide NSTDATA_API_TOKEN plus APPROVED_PAGE_URL. The example below is credential-dependent and was not run with a customer token during preparation of this guide.
Step 2: Validate one page through the SDK
Use the request and completion checks below before introducing discovery. Its purpose is to establish the collection-and-validation boundary for one known page.
import os
from pathlib import Path
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto
token = os.environ["NSTDATA_API_TOKEN"]url = os.environ["APPROVED_PAGE_URL"]with NstDataClient(token)as client: result = client.submit_scrape_task_sync(ScrapeRequestDto( url=url, formats=["markdown"], onlyMainContent=True, timeout=60000))if result.success isnotTrueor result.status !="completed":raise RuntimeError("Page retrieval was not completed successfully")if result.data isNoneor result.data.truncated isTrue:raise RuntimeError("Missing or partial content requires review") text = result.data.get_markdown()ifnot text ornot text.strip():raise RuntimeError("No usable Markdown returned") Path("page.md").write_text(text, encoding="utf-8")print("Saved page.md; validate its content before downstream use")
Step 3: Add site discovery after acceptance
Use the current SDK's bounded crawl operation or copy the generated site-crawl example from the Playground. Preserve the returned crawl ID, poll the documented status operation, and retrieve all result pages using its pagination contract. A single result page should not be mistaken for the complete job output.
What Does Nstdata Crawl Manage, and What Do You Still Own?
Nstdata Crawl manages the collection layer behind a Web Crawling API for LLM-ready data. Its product surface supports site discovery, recursive crawling, JavaScript rendering, and multiple output representations. A developer can use the API rather than assembling every collection component independently. Built-in proxy access can serve a standard managed request without a separate gateway configuration. Your application still owns rights decisions, URL relevance, schema validation, storage, and the meaning of completeness.
Discovery and recursion: Bound the seed with page and depth limits. Treat the discovered set as a measured output rather than assuming that every page of the site was reached.
Output flexibility: Markdown supports model context, HTML supports parser inspection, links support graph analysis, and JSON, PDF, or visual artifacts serve other delivery and review needs.
Managed access: Use the built-in proxy option without separate proxy credentials for the standard managed path. Check current billing, including proxy usage, before extrapolating pilot costs.
Operational evidence: Keep task identifiers and page outcomes with the collection manifest. Separate a completed job from a dataset that passed your acceptance rules.
Why Do Crawlers Get Stuck or Produce Incomplete Datasets?
Crawlers get stuck or produce incomplete datasets when discovery grows without a stopping rule, URL identity is inconsistent, or retrieval success is confused with content acceptance. A crawler can finish its queue correctly while still missing pages that were never discoverable from the configured seeds.
Symptom
Cause
Useful correction
URL count grows indefinitely
Calendars, filters, or query combinations
Constrain paths and meaningful parameters
Many repeated pages
Weak canonicalization or redirect handling
Define identity and record redirect relationships
Empty text on dynamic templates
Required data appears after initial HTML
Inspect rendering and wait conditions
Important pages absent
Orphans, excluded paths, or shallow depth
Compare with an independent expected inventory
High request success, low useful output
Error pages accepted as content
Add page-type and required-content checks
Build on sitemap-aware discovery and retain the last successful checkpoint. Retry transient network failures with a bounded policy; investigate authorization failures instead of repeatedly resubmitting them.
How Should You Judge Whether a Crawler Is Working?
Judge a crawler by coverage of the intended scope, correctness of accepted content, freshness, and cost per useful page. Request count and job completion are operational metrics, but neither measures the dataset's fitness for your product.
Use production retry handling to keep temporary failures distinct from terminal outcomes. Start with a small permitted site section and an expected-URL sample, inspect what the crawler actually returns, and expand only after the failure categories are understood. Nstdata's alternate HTML and visual outputs can help investigate extraction mismatches before those pages enter an AI knowledge source.
A crawler primarily discovers and retrieves pages, while a scraper extracts information. Many tools combine the stages, but the responsibilities are still distinct.
Q: Does a sitemap guarantee complete coverage?
No. A sitemap supplies candidate URLs and can be incomplete or stale. Compare the discovered set with other approved inventories when completeness matters.
Q: Do all pages require a browser?
No. Use rendering when the required information is absent from the initial response. A browser adds cost and failure modes that a simple retrieval may not need.
Q: Is robots.txt permission to reuse content?
No. Robots rules communicate crawler instructions; access rights, privacy, copyright, and reuse permissions require their own assessment.
Q: What does a completed crawl prove?
A completed crawl proves that the job reached its terminal state under its configuration. Validate page outcomes and expected coverage before calling the dataset complete.
Lena Zhou
Nov. 9th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more β without managing crawling infrastructure.
