How to Build a RAG Knowledge Base From Any Website 2026
TL;DR
A useful RAG knowledge base starts with clean, traceable source documents—not an embedding model. Preserve canonical URL, title, crawl time, headings, and content hash with every chunk.
The pipeline is crawl → normalize → deduplicate → chunk → embed → store → retrieve → answer with citations. Evaluate each boundary independently so retrieval errors are not mistaken for model errors.
Markdown is a practical interchange format for website ingestion. It removes much of the navigation noise while keeping heading structure that can guide semantic chunking.
Use Nstdata Crawl when you need rendered website content without maintaining browsers and proxy orchestration. The example below uses its current scrape route, then demonstrates a dependency-free local retrieval baseline.
A retrieval-augmented generation system is only as trustworthy as the evidence it can retrieve. If website ingestion stores cookie banners, repeated navigation, stale copies, and chunks with no source metadata, a stronger language model cannot repair the missing provenance.
This tutorial builds a small, inspectable pipeline and shows where production systems need stronger components. It uses a local hashed-vector baseline so the retrieval logic can run with Python’s standard library. Replace that baseline with a production embedding model and vector index after the data contracts are working.
What Does a Crawl-for-RAG Pipeline Need?
A crawl-for-RAG pipeline needs six properties: coverage, clean content, stable identity, useful segmentation, retrievable vectors, and source-grounded answers. Speed matters, but completeness and traceability matter more.
The minimum record for each fetched page should contain:
Field
Why it matters
source_url
Lets an answer cite evidence
canonical_url
Prevents URL-parameter duplicates
title and headings
Improves display and semantic boundaries
crawled_at
Supports freshness policies
content_hash
Detects unchanged or duplicate content
markdown
Provides normalized text with structure
http_status
Separates unavailable pages from empty extraction
locale/access context
Explains regional or language differences
Nstdata Crawl can turn URLs into formats such as Markdown and JSON while handling rendering infrastructure. It does not replace your knowledge-base policy: your application must still decide which paths are allowed, how frequently they refresh, and what qualifies as an accepted document.
Before crawling, inspect the site’s rules and your authorization. The Robots Exclusion Protocol defines how crawlers discover robots.txt instructions, but robots compliance is only one part of legal and contractual review. See Nstdata’s web scraping legal guide for a broader checklist.
Step 1: Define Scope and Crawl the Website
Start with a sitemap, a known documentation root, or a curated URL seed. Add explicit allow and deny rules. For a docs site, you may allow /docs/ and exclude login, search, changelog pagination, query parameters, and downloadable binaries.
Nstdata’s live API currently accepts authenticated scrape requests at the route below. The route was verified to reach authentication on September 3, 2026; a successful response could not be tested without an account key. Confirm request fields in the current Crawl documentation before production use.
Store the raw response before normalization. Raw retention makes parser upgrades reproducible and gives you evidence when an extraction rule changes. Never log API keys, cookies, or personal data from authenticated pages.
If you are deciding between discovery methods, read scraping vs crawling. Scraping one known URL is appropriate for targeted refreshes; crawling is appropriate when link discovery is part of the job.
Step 2: Normalize and Deduplicate Before Embedding
Normalization should remove repeated navigation, footer text, cookie notices, invisible controls, and empty headings without flattening meaningful document structure. Markdown helps because headings, lists, code, and links survive in a compact form. CommonMark provides a useful baseline for consistent Markdown parsing.
Use two identities:
URL identity: normalize scheme/host casing, strip tracking parameters, resolve redirects, and prefer an authoritative canonical URL.
Content identity: hash normalized body text to detect mirrored or parameterized duplicates.
Do not discard every near-duplicate blindly. Product documentation may repeat a shared warning but contain different procedures. Exact hashes are safe for exact duplication; fuzzy similarity should produce reviewable candidates.
Freshness belongs in the same contract. Store crawled_at, source modification hints when available, and the extractor version. On refresh, re-embed only changed chunks and delete vectors for removed pages.
Step 3: Chunk Around Meaning, Not Arbitrary Character Counts
A chunk should be large enough to answer one likely question and small enough to retrieve precisely. Begin with heading-aware segments, then split long sections by paragraphs or sentences. Attach breadcrumb headings to every child chunk.
A practical starting policy is:
split at H2/H3 boundaries;
target roughly 300–700 tokens per chunk;
keep code blocks and tables intact when possible;
add a small overlap only across genuinely continuous prose;
prepend the page title and heading path to the embedded text;
store the unmodified display text separately.
There is no universally best chunk size. Evaluate on real questions. If answers need facts scattered across a long procedure, retrieve neighboring chunks or use parent-child retrieval instead of making every chunk enormous.
Step 4: Embed, Store, and Retrieve a Runnable Local Baseline
The following script implements the complete local mechanics with only Python’s standard library. It uses a deterministic hashed bag-of-words vector—not a semantic production embedding. That limitation is deliberate: you can run the data flow locally, inspect SQLite records, and later replace only embed().
import hashlib
import json
import math
import re
import sqlite3
from datetime import datetime, timezone
DIMENSIONS =256defnormalize(text:str)->str: text = re.sub(r"\r\n?","\n", text) text = re.sub(r"[ \t]+"," ", text) text = re.sub(r"\n{3,}","\n\n", text)return text.strip()defchunk_markdown(markdown:str, max_words:int=90): chunks, heading,buffer=[],"",[]for line in normalize(markdown).splitlines():if line.startswith("#"):ifbuffer: chunks.append((heading,"\n".join(buffer)))buffer=[] heading = line.lstrip("# ")else:buffer.append(line)iflen(" ".join(buffer).split())>= max_words: chunks.append((heading,"\n".join(buffer)))buffer=[]ifbuffer: chunks.append((heading,"\n".join(buffer)))return[(h, t.strip())for h, t in chunks if t.strip()]defembed(text:str): vector =[0.0]* DIMENSIONS
for token in re.findall(r"[a-z0-9]+", text.lower()): slot =int(hashlib.sha256(token.encode()).hexdigest()[:8],16)% DIMENSIONS
vector[slot]+=1.0 length = math.sqrt(sum(x * x for x in vector))or1.0return[x / length for x in vector]defcosine(a, b):returnsum(x * y for x, y inzip(a, b))defingest(db, url, title, markdown): cleaned = normalize(markdown) page_hash = hashlib.sha256(cleaned.encode()).hexdigest() crawled_at = datetime.now(timezone.utc).isoformat() db.execute("DELETE FROM chunks WHERE source_url = ?",(url,))for index,(heading, text)inenumerate(chunk_markdown(cleaned)): embedding_text =f"{title}\n{heading}\n{text}" db.execute("INSERT INTO chunks VALUES (?, ?, ?, ?, ?, ?, ?)",(url, title, heading, index, text, json.dumps(embed(embedding_text)),f"{page_hash}:{crawled_at}"),) db.commit()defsearch(db, question, limit=3): query_vector = embed(question) rows = db.execute("SELECT source_url, title, heading, body, vector FROM chunks").fetchall() ranked =[(cosine(query_vector, json.loads(vector)), url, title, heading, body)for url, title, heading, body, vector in rows
]returnsorted(ranked, reverse=True)[:limit]db = sqlite3.connect(":memory:")db.execute("""CREATE TABLE chunks (
source_url TEXT, title TEXT, heading TEXT, chunk_index INTEGER,
body TEXT, vector TEXT, version TEXT
)""")sample ="""# Acme Docs
## Authentication
Send an API key in the Authorization header. Never expose the key in client code.
## Retries
Retry rate limits with exponential backoff and jitter. Do not retry invalid credentials.
"""ingest(db,"https://example.com/docs","Acme Docs", sample)for score, url, title, heading, body in search(db,"How should I handle rate limits?"):print(f"{score:.3f}\t{heading}\t{url}\t{body}")
The script was executed locally with Python 3 using the included illustrative document. It ranked the “Retries” section first for the rate-limit question. This confirms chunk storage and retrieval wiring; it does not validate semantic quality on a real corpus.
For production, swap the local vector function for an embedding API, and replace the linear scan with a vector index. pgvector adds exact and approximate vector search to PostgreSQL; managed vector databases are another option. Preserve the same metadata regardless of storage engine.
Step 5: Generate Answers Only From Retrieved Evidence
Pass the best chunks to the answer model with source URLs and a strict instruction: answer from the supplied context, cite claims, and say when evidence is insufficient. Do not let the model silently substitute general knowledge for missing site content.
A robust answering stage should:
apply an absolute relevance threshold, not just “top three”;
diversify results so one duplicated page does not occupy every slot;
include neighboring chunks for procedures;
filter by tenant, locale, product version, and access control;
cite the canonical source URL beside each supported claim;
log retrieved chunk IDs for later evaluation.
Reranking can improve precision after initial vector retrieval. Hybrid retrieval—semantic vectors plus keyword scoring—is particularly useful for error codes, product names, and exact API parameters.
Nstdata’s web search MCP server guide offers related context for connecting live web evidence to AI agents. A fixed knowledge base and live search solve different problems: the former is controllable and fast, while the latter can discover newer pages.
Step 6: Evaluate the Pipeline End to End
Build a question set from real support tickets, documentation headings, and known failure cases. For each question, label the expected source page and whether the corpus contains an answer.
freshness lag: time from source change to searchable update;
retrieval recall: expected evidence appears in the candidate set;
citation precision: cited pages actually support the answer;
answer faithfulness: claims are entailed by retrieved text;
abstention quality: the system refuses when evidence is absent.
Debug in that order. If the right page was never crawled, tuning embeddings is wasted effort. If the right chunk was retrieved but the answer ignored it, change the prompt or model stage.
Common RAG Ingestion Mistakes
The most common mistake is embedding raw HTML. It fills the index with menus, script text, and repeated template content. Other expensive errors include chunking before deduplication, losing heading paths, omitting canonical URLs, refreshing the whole corpus instead of changed pages, and allowing deleted content to remain searchable.
Security is equally important. Do not crawl private pages into a shared index unless retrieval enforces the source authorization model. Treat scraped text as untrusted input: it can contain prompt injection aimed at downstream agents. Keep system instructions separate and restrict what answer-time tools can do.
For a broader provider comparison before committing, use Nstdata’s best web scraping API guide. Then test Nstdata Crawl on your hardest pages, not only a static homepage.
Production Checklist
Before launch, confirm that every chunk has a canonical source, heading path, crawl timestamp, version, and content hash. Verify deletion propagation, access filters, retry limits, observability, and a documented refresh policy. Keep a small golden question set in CI so parser or model changes cannot silently reduce retrieval quality.
The crawler is the first component, but it sets the ceiling for everything downstream. Clean, versioned Markdown plus explicit quality gates gives the embedding and answer stages evidence they can actually use.
Crawl for RAG is the process of discovering website pages, extracting clean content and metadata, chunking and embedding it, and storing it for retrieval-augmented generation.
Q: Should I store HTML or Markdown for RAG?
Markdown is usually easier to chunk and audit because it removes much page chrome while preserving headings, lists, links, tables, and code. Retain raw HTML separately when reprocessing or compliance requires it.
Q: How often should a RAG knowledge base be refreshed?
Refresh frequency should follow source volatility and business risk. A documentation site may need event-driven or daily updates; a stable archive may need less. Use content hashes to reprocess changed pages and remove deleted content promptly.
Q: Does the example use production embeddings?
No. It uses a deterministic local hashed vector to demonstrate the pipeline without dependencies. Replace it with a semantic embedding model and a vector index before production use.
Ivy Lin
Sep. 4th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
