What Does a Web Data Pipeline for an AI Product Produce?
A useful pipeline produces a versioned document or structured record that the product can trace back to a source. For RAG, the output might be Markdown split into chunks with source URLs and heading paths. For an extraction feature, it might be a typed JSON object with evidence for each field. The pipeline should also produce failure records: a timeout, blocked page, or rejected document is an outcome to investigate, not a reason to silently disappear a URL. The Nstdata Crawl product is a managed option for the acquisition and conversion steps discussed here.
The work begins before fetching. Identify permitted sources, target page types, update expectations, and the question the AI product must answer. A support agent trained on product documentation needs a different refresh policy from a price-monitoring feature. The Databricks RAG pipeline guide separates document preparation from indexing and serving; keep those boundaries visible in your own design.
Prepare an approved source list; one static page, one dynamic page, and one error case; an Nstdata account and server-side Crawl API key if using managed acquisition; and a store for both accepted records and job outcomes. Define a stable key such as source_id + canonical_url + extractor_version. A content hash detects changes, but it is not a substitute for URL identity: two pages can contain the same boilerplate while representing different products.
For Nstdata Crawl, the documented request uses x-api-key, Content-Type: application/json, url, and formats. Site discovery also requires explicit maxDepth, maxPages, includeUrls, and excludeUrls. Standard managed jobs use built-in proxy routing, so no separate Host, Port, Username, Password, or Protocol configuration is required. Check current account limits and source access rules before scaling.
How Do You Build the First Working Pipeline?
Method 1: Prototype with a known URL
Use this path when an application already knows the page to retrieve.
Step 1: Define an acceptance fixture
Record the expected title, two distinctive phrases, required links, and source URL for one approved page. Decide which missing item rejects the document.
Step 2: Fetch Markdown and HTML
The Crawl documentation shows a single-page quick-start route. Store the key in NSTDATA_API_TOKEN and submit a representative page:
curl--output response.json --request POST \"https://api.nstdata.io/api/v1/crawl/scrape/submit-sync"\--header"x-api-key: ${NSTDATA_API_TOKEN}"\--header"Content-Type: application/json"\--data'{"url":"https://example.com/","formats":["markdown","html"],"onlyMainContent":true}'
The public docs contain conflicting labels on some other synchronous and asynchronous scrape routes; use the documented quick start and compare the request generated by your Playground before production deployment.
Step 3: Validate the nested result
Check data.success, data.status, and data.data.metadata.statusCode before accepting markdown. If the response contains markdownRef or htmlRef, retrieve the exact token with GET /api/v1/crawl/storage/read?st={ref}. Compare the body with your fixture; do not treat outer HTTP 200 as proof of a useful page.
Method 2: Discover a bounded site section
Use this path when the URL list itself must be collected.
Step 1: Set boundaries
Choose a start URL, maxDepth, maxPages, includeUrls, and excludeUrls. Exclude search, account, and query-generated paths unless they are part of the product scope.
Step 2: Submit the crawl
Call POST /api/v1/crawl with formats:["markdown"], save data.id, then poll GET /api/v1/crawl/status/{crawlId}. A site job discovers links recursively; it is not an unlimited bulk-URL endpoint.
Step 3: Reconcile results
Retrieve GET /api/v1/crawl/pages/{crawlId} and follow returned pagination. Store submitted, completed, failed, rejected, and accepted counts separately. Compare the discovered URL set with any sitemap or source inventory you already maintain.
Method 3: Keep acquisition local
Use a local HTTP client when sources are stable, static, and within your operational capacity. Fetch with a timeout, per-host pacing, and a response-size cap. Parse the saved HTML, then apply the same acceptance fixture and storage contract as the managed path. This option gives you direct control but makes your team responsible for browser rendering, proxy selection, retries, and observability when the source changes.
Write an immutable raw acquisition object and a separate accepted document. The accepted record should contain source_url, final_url, fetched_at, status_code, content_hash, format, extractor_version, and a pointer to the raw artifact. For RAG, also store chunk IDs, heading paths, and index version. Index only accepted records; a failed refresh should not replace a good document with an empty one.
Make publishing to the search index idempotent. If a document hash has not changed, skip re-embedding unless the chunker or embedding model changed. If a page is removed, record a tombstone and remove its chunks from serving indexes. Keep source and index timestamps separate so an answer can disclose how current its evidence is. The Azure RAG design guide emphasizes evaluation across ingestion, retrieval, and answer quality.
Run the fixture set after every parser or configuration change. Track accepted records per submitted URL, missing required fields, duplicate canonical URLs, time from source change to index update, and cost per accepted record. Sample outputs from each template, not only the easiest pages. For an AI product, add answer-level evaluation: can the retriever locate the right document, and does the model cite the source span that supports its answer?
Nstdata Crawl can reduce acquisition work, but it does not choose your field schema or assess an answer. Use Cloud Browser for workflows that need continuing page interaction rather than page-to-data conversion. Keep these product boundaries clear when designing the pipeline.
Give every stage a durable input and output identity. Discovery should emit a URL manifest with source policy and run ID. Acquisition should save a raw artifact and a terminal outcome for each URL, including failures. Transformation should reference the artifact ID and parser version. Indexing should reference the accepted document version. With these links, a worker can retry its own stage without replaying every earlier stage.
For a multi-site product, partition the manifest by source. A documentation site, a product catalog, and a news feed have different update rates and page templates. Keep per-source quotas and error budgets so a broken site does not consume the whole queue. A source owner should be able to pause one partition without stopping valid collection elsewhere.
Use at-least-once processing with idempotent writes when perfect exactly-once execution is impractical. A worker may finish writing a document and crash before acknowledging its queue message. On restart, it will see that message again. A stable key such as source_id + canonical_url + content_hash + parser_version lets it recognize the same version. Do not deduplicate solely on content hash: two legitimate pages can contain identical short text while having different source meanings.
Set a terminal state for failures. retry_pending should have a next eligible time and attempt count. rejected should have a content-quality or policy reason. failed_terminal should preserve the last error. A dashboard that counts only complete leaves the team unable to reconcile the URL manifest with delivered documents.
How Do You Prevent Bad Data from Reaching the Index?
Use a promotion step rather than writing a fetch result directly to serving storage. Compare the candidate with the previous accepted version. Check expected page type, title, minimum relevant content, language, required fields, and suspicious changes in length. A 95% drop in text length may be a valid rewrite, but it deserves inspection before an automated replacement. For structured records, validate types, currencies, units, and entity identity rather than just JSON syntax.
Keep a last-good version and mark its age accurately. If a refresh produces a login page or error shell, the current product should either show the older record with a stale indicator or suppress it according to its use case. It should not publish the failed capture as current content. That same promotion step is where policy filtering and personal-data minimization belong.
For RAG, test retrieval after promotion. A successful embedding job does not prove that the right chunk will be returned. Maintain a small set of questions whose answers map to known source spans; compare retrieval and citations after each index change. This separates an extraction failure from a chunking or ranking failure.
For structured extraction, keep the evidence span for each field. If a price changes unexpectedly, a reviewer can inspect the exact page text or rendered capture that produced it. This is more useful than a JSON object with no provenance. The same record can be revalidated when business rules change, and bad fields can be corrected without treating the whole crawl as unusable.
No. Index only pages that pass status, content, scope, and semantic checks.
Q: Can a successful refresh make the product worse?
Yes. A newly fetched consent wall or incomplete page can replace good content unless validation blocks promotion.
Q: What is the first useful scale test?
Test several templates and failure cases under the same acceptance contract before increasing page volume.
This Python tutorial moves from one static page to validated records and managed rendering. It includes runnable examples, failure diagnosis, source controls, and an AI-ready data contract.
Marcus Chen
Oct. 29th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
