Β©2026 NST LABS TECH LTD. All RIGHTS RESERVED.
A crawl API supplies pages; a RAG pipeline needs accepted, normalized, source-bearing documents.
Raw HTML carries navigation, scripts, repeated UI, and unstable markup that can pollute chunks and retrieval.
Convert and validate content, then chunk by structure, embed with a versioned model, and retain canonical URL and crawl time.
Measure answer support and citation quality after indexing, not just crawler response success.
Compare Firecrawl vs Nstdata Crawl on features, pricing, JavaScript rendering, scalability, and migration costs. Learn when and how to switch.
Learn how to clean, validate, chunk, and index website content for a source-aware RAG pipeline.
Test Nstdata Crawl on your own URLsInspect accepted page artifacts and complete billing before scaling. Try Nstdata Crawl |
Markdown
JSON
{
Β Β "title": "...", Β Β "url": "..." } Screenshot
|
You can pass HTML text to a model, but that is rarely a good retrieval corpus. A page often includes menus, cookie notices, related links, scripts, hidden text, and duplicated templates. If you split by arbitrary character count, chunks can cross headings and tables or repeat the same navigation hundreds of times. Retrieval then ranks page chrome alongside the answer. Nstdata Crawl offers Markdown and other output artifacts, which is a useful starting point, but format conversion is only one quality gate.
Keep enough provenance to debug a bad answer: original URL, canonical URL, retrieval timestamp, page status, title, output format, crawl job ID, and a raw or recoverable artifact reference. Nstdata's Crawl documentation separates site-job status, page records, and storage reads. If a record contains markdownRef, fetch the referenced body before chunking. Do not treat a completed job as proof that every page is usable. A 403 page that says βAccess Deniedβ can be syntactically valid Markdown and semantically useless.
The Scrapy overview shows the alternative of owning the crawler. The output contract should be an application-level document, not a vendor response. A minimal record might be {id, canonical_url, title, markdown, fetched_at, status, raw_ref}. Do not put a storage reference into the markdown field. The Firecrawl crawl guide likewise separates a crawl job from returned page data; both vendors need a normalizer before RAG indexing.
A chunk should be readable without the rest of the page but should retain its heading path and source URL. Keep code blocks intact where possible; split long tables by rows with repeated column headings; avoid stitching unrelated FAQ answers together. Overlap can help when a sentence crosses boundaries, but too much overlap increases index size and duplicate retrieval. Store the chunker version because changing split rules requires an index migration or reprocessing.
A useful quality sample includes pages with nested headings, code examples, tables, images with captions, and PDF-like layouts. Inspect the actual Markdown output, not only a rendered preview. If target pages include JavaScript content, check whether the crawler waits for the relevant body. The Crawl4AI documentation is a useful reference for extraction and rendering controls when you compare build and buy options.
Evaluate three layers. Acquisition: what share of expected URLs produced accepted documents, and what were the failure classes? Retrieval: for fixed questions, do the correct chunks appear in the top results? Answering: are claims supported by cited URLs, and does the model abstain when no evidence exists? Use a held-out question set and repeat after recrawls. A high crawl response count can coexist with a poor answer set if navigation or stale duplicates dominate the index.
Record full cost per accepted document and per successful answer. Nstdata's Crawl pricing page lists URL rates plus proxy traffic, while its product FAQ has a conflicting lower figure; confirm the current account rate. The RAG knowledge-base guide and crawl success-rate methodology give a path for measuring the end-to-end result rather than assuming the crawler solved it.
Use one when site discovery, rendered retrieval, and page artifacts would otherwise absorb scarce engineering time. Keep your own quality gates, metadata model, and index update logic so the corpus remains portable. If you already have a trustworthy URL list and only need page fetches, compare a fetch API too. The right contract is the one that reliably produces your accepted document, with a cost and error model your team can operate.
Inspect a bounded Nstdata crawl on your own permitted site, then validate each artifact against your RAG document contract.
Q: Is Markdown automatically LLM-ready?
No. It can still contain boilerplate, blocked-page text, duplicates, and stale content. Validate it before indexing.
Q: Should I store raw HTML too?
Retaining a raw or recoverable artifact helps debug extraction changes and disputed answers, subject to your storage policy.
Q: What metadata should every chunk have?
At minimum, keep canonical URL, title or heading path, crawl time, stable ID, and content or chunker version.
Q: How often should I recrawl?
Base the schedule on how often the source changes and how costly stale answers are; monitor changed and removed pages.
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
