What does a website RAG pipeline need to preserve?
A website corpus is not a bag of Markdown strings. Each page needs source URL, canonical URL, crawl time, title, content, and a stable ID. Nstdata Crawl documents site crawling and multiple outputs, while its Crawl API guide describes status, page listing, and artifact retrieval. The crawler is the acquisition stage; your application owns acceptance, chunking, embedding, indexing, and citation checks. Use a bounded, permitted site for the first run.
The code below is a fully runnable offline demonstration with Python standard library. Its fixture stands in for the normalized output of a crawl adapter. The hash-based vector is intentionally a deterministic test double; it is not a semantic embedding and should never be used to judge answer quality. A production adapter would poll a Nstdata crawl, retrieve Markdown references, and pass Page objects into the same downstream functions. The public docs disagree on some scrape route labels, so confirm the live route before implementing that network step.
Full crawl-to-query example
Save this as rag_demo.py and run python3 rag_demo.py. It prints the source of the highest-scoring chunk for a sample query.
from dataclasses import dataclass
from hashlib import sha256
from math import sqrt
import re
@dataclass(frozen=True)classPage: url:str title:str markdown:str fetched_at:strpages =[ Page("https://example.com/docs/auth","Authentication","# Authentication\nUse an API key in the Authorization header.\n""Rotate keys if a credential is exposed.","2026-10-16T00:00:00Z"), Page("https://example.com/docs/limits","Rate limits","# Rate limits\nBack off on HTTP 429 and inspect Retry-After.\n""Use bounded concurrency for batch jobs.","2026-10-16T00:00:00Z"),]defchunks(page: Page, max_words:int=35): words = page.markdown.split()for offset inrange(0,len(words), max_words): body =" ".join(words[offset:offset + max_words]) chunk_id = sha256(f"{page.url}:{offset}:{body}".encode()).hexdigest()[:16]yield{"id": chunk_id,"url": page.url,"title": page.title,"fetched_at": page.fetched_at,"text": body}defdemo_vector(text:str, dimensions:int=64): vector =[0.0]* dimensions
for token in re.findall(r"[a-z0-9]+", text.lower()): slot =int.from_bytes(sha256(token.encode()).digest()[:4],"big")% dimensions
vector[slot]+=1.0 norm = sqrt(sum(value * value for value in vector))or1.0return[value / norm for value in vector]index =[(chunk, demo_vector(chunk["text"]))for page in pages for chunk in chunks(page)]query ="Rate limits: back off on HTTP 429"needle = demo_vector(query)ranked =sorted(index, key=lambda item:sum(a * b for a, b inzip(needle, item[1])), reverse=True)for chunk, _ in ranked[:2]:print(chunk["title"], chunk["url"], chunk["text"])
I executed this offline fixture using Python's standard library and confirmed it returns ranked, source-bearing chunks. The test proves wiring, not semantic relevance or crawl quality. Replace demo_vector with a versioned embedding model and store vectors plus metadata in a database before deploying. Keep the model version in the index; changing embeddings requires re-embedding or an isolated collection.
Choosing a vector store without premature complexity
For a small corpus or prototype, a local index or database vector extension can be enough. A managed vector database becomes useful when the team needs larger indexes, replica management, filters, or operational separation. The Pinecone documentation describes one hosted vector database option. Pinecone is one option; this crawl-to-Pinecone walkthrough covers an integration path. The selection criteria are filter support, update/delete semantics, dimension compatibility, observability, and cost at your expected chunk count. Do not choose only by maximum advertised vectors.
The chunker above splits by word count for clarity. Production code should preserve headings, code fences, tables, and links. Nstdata's chunking guide explains alternatives. A safe pipeline should reject empty or error pages, deduplicate canonicals, generate deterministic chunk IDs, upsert changed chunks, and delete chunks for removed pages. Track crawl ID and embedding version alongside every chunk.
How to make answers trustworthy
Retrieve top chunks with metadata filters, then ask the language model to answer only from those chunks and cite their URLs. The answer generator should say when retrieved evidence is insufficient. Evaluate a fixed set of questions with expected supporting URLs, including negative questions that the site cannot answer. Measure citation correctness, answer support, coverage, and freshness after recrawls. The website RAG guide gives more of the operational workflow.
For a managed acquisition comparison, Firecrawl's crawl documentation and the Crawl4AI documentation show alternative ways to supply the same normalized Page records. The downstream contract should stay stable. If the answer quality changes after swapping crawlers, inspect page acceptance and chunk content before blaming the embedding model.
Experience Nstdata — Start Your Free Trial Today
Start with a bounded crawl, inspect each Markdown artifact, and connect accepted pages to your own index adapter.
