Feeding Your RAG System with Fresh Web Data (Step by Step)
TL;DR
A RAG ingestion pipeline needs a source-of-truth crawl, a stable document identity, and a refresh path; embeddings alone do not keep answers current.
Use a site crawl for bounded documentation discovery, then validate each page before cleaning, chunking, embedding, and indexing it.
Store source_url, crawled_at, content_hash, and a stable page ID with every chunk so updates and citations remain manageable.
Treat the code below as an integration blueprint: the API fields follow current Nstdata and library documentation, but the end-to-end path was not run against live credentials.
Introduction: Fresh web data is an ingestion problem
Nstdata Crawl is the collection layer in this workflow; its product overview describes page and site retrieval for downstream data pipelines.
A language model can explain an API from its training data and still give you an obsolete parameter name. A RAG system narrows that gap by retrieving documents at answer time, but retrieval is only as current as the documents already indexed. To feed a RAG system with fresh web data, you need a repeatable path from source URLs to validated records and a way to replace or remove old records.
This guide builds that path for public documentation: Nstdata Crawl → Markdown validation → heading-aware chunks → embeddings → Chroma → retrieval. It focuses on ingestion and freshness, not on training a model. Nstdata Crawl handles page acquisition and content conversion; your application still owns data policy, chunking, indexing, access control, and answer evaluation. The current Crawl guide describes Markdown output, page/site tasks, Python SDKs, task status, and large-result references in its Crawl documentation.
What RAG data ingestion means
RAG data ingestion is the process that turns source material into records a retrieval system can search. Model training data is baked into model weights. A RAG index is an external, replaceable collection of documents and embeddings. Live web data is what your collector retrieves from a site now; it does not reach the model until your pipeline validates and indexes it.
If a docs page changes at 10:00 but your index is rebuilt weekly, a query at 10:05 may still retrieve last week’s instructions. The answer can sound fluent and still be stale. Record both when the source was collected and which content version was indexed to make that delay visible.
Pipeline at a glance
Stage
Input → output
Main check
Discover
domain, sitemap, or URL list → allowed URLs
Are URLs in scope and permitted?
Crawl
URL → Markdown and page metadata
Did the task complete and does the body contain expected content?
Normalize
fetched page → canonical document record
Are URL, title, version, and content hash stable?
Chunk
document → retrievable passages
Did headings, code, and tables stay intact?
Embed and store
passages → vectors plus metadata
Do vector dimension and record IDs match the index?
Retrieve and answer
question → evidence and cited answer
Can the answer be traced to current source passages?
Prerequisites and the example stack
Use Python 3.10 or later, an Nstdata Crawl token, an embedding provider key, and a Chroma collection. Install the SDKs and lock the resolved versions in your project after a staging run:
The exact versions should be recorded in your lockfile because SDK behavior can change. The environment used to prepare this draft has no Crawl or OpenAI credentials, so the service calls below have not been executed live.
Step 1: Select and discover web sources
Start with a short inventory rather than crawling an entire domain. For a docs site, include the current product and API reference pages, then exclude login, search, changelog archives, and unrelated marketing paths if they are outside the knowledge base. Use a sitemap and navigation links together when you need to compare declared URLs with reachable pages. A sitemap is a discovery hint, not proof that every listed page is useful or current.
Use a single-page scrape when a known URL is the whole input. Use site-wide Crawl when you want bounded discovery from a starting page. Current Crawl requests expose maxDepth, maxPages, includeUrls, excludeUrls, ignoreQuery, and onlyMainContent; choose conservative boundaries and inspect the discovered page list before indexing. For a separate walkthrough of the vector-database side, see Crawl to Pinecone.
Step 2: Crawl a page into Markdown
The documented Python SDK exposes submit_scrape_task_sync and get_markdown(). This request is suitable for a known URL with a predictable response time:
import os
from nstdata_ai_crawl import Format, NstDataClient, ScrapeRequestDto
with NstDataClient(os.environ["NSTDATA_API_TOKEN"])as client: result = client.submit_scrape_task_sync(ScrapeRequestDto( url="https://docs.example.com/api/authentication", formats=[Format.MARKDOWN], timeout=60_000, onlyMainContent=True,))markdown = result.data.get_markdown()ifnot markdown orlen(markdown.strip())<200:raise ValueError("Crawl returned empty or unexpectedly short content")
A successful HTTP exchange is not the same as a usable page. Check task success, body length, expected headings or a known phrase, and returned metadata before you mark the page accepted. For slower pages, use the asynchronous task flow and poll the returned ID; the official guide also documents page progress and result references. Do not assume a webhook callback exists: the current docs say webhooks are not supported.
For JavaScript-rendered pages, first inspect whether the required text is missing from the normal output. Only enable browser rendering or a documented wait/action when a page actually needs it. Extra rendering adds latency and can make a crawl more complex to diagnose.
Step 3: Normalize and preserve provenance
Before chunking, turn the fetched result into a stable record. Keep source metadata separate from body text:
from dataclasses import dataclass
from datetime import datetime, timezone
from hashlib import sha256
@dataclassclassPageRecord: url:str title:str markdown:str crawled_at:str content_hash:strdefmake_record(url:str, title:str, markdown:str)-> PageRecord: normalized ="\n".join(line.rstrip()for line in markdown.splitlines()).strip()return PageRecord( url=url, title=title or url, markdown=normalized, crawled_at=datetime.now(timezone.utc).isoformat(), content_hash=sha256(normalized.encode("utf-8")).hexdigest(),)
A content hash lets the pipeline skip unchanged pages. Normalize only noise that is truly irrelevant: preserve headings, code fences, parameter tables, warnings, and source links. Removing boilerplate is not permission to rewrite the author’s facts.
Step 4: Chunk around document structure
For technical docs, split on headings before applying a token or character limit. A chunk should carry its heading path so a retrieved paragraph still has context:
defheading_sections(markdown:str): current =[] body =[] in_fence =Falsefor line in markdown.splitlines():if line.strip().startswith("```"): in_fence =not in_fence
if line.startswith("#")andnot in_fence and body:yield" / ".join(current),"\n".join(body).strip() body =[]if line.startswith("#")andnot in_fence: level =len(line)-len(line.lstrip("#")) current = current[:level -1]+[line[level:].strip()] body.append(line)if body:yield" / ".join(current),"\n".join(body).strip()
This small splitter keeps fenced examples together but does not enforce a token ceiling. Add a tokenizer-aware second pass for long sections and test it against your longest code sample. Framework users can compare that baseline with the current LangChain text-splitter concepts. Avoid guessing a universal chunk size: compare retrieval on questions that require parameter details, cross-section context, and code.
Step 5: Embed and store records
Create an embedding per chunk and write the text and metadata with it. Chroma can persist a local collection; its upsert and query APIs are documented in the Chroma collection guide. The provider model and Chroma embedding dimension must agree if you pass precomputed vectors. The current OpenAI embeddings guide documents the text-to-vector endpoint used in this example.
In production, send embeddings in batches, retry only transient provider errors, and cap batch size according to provider limits. Make IDs deterministic, for example sha256(canonical_url + heading_path + chunk_text). When a page changes, delete the old page’s chunk IDs before upserting its new set, or use a versioned namespace and switch over after validation.
Step 6: Retrieve and answer with evidence
Embed the user’s question with the same model, query the collection, and pass the top passages plus source URLs to the LLM. Require the answer to cite the supplied URLs and to say when the retrieved evidence is insufficient. Retrieval output is evidence to inspect, not proof that the answer is correct.
A useful evaluation set includes direct facts, API parameter questions, questions whose answer spans two sections, and questions that should return “not found.” Track retrieval hit rate and citation correctness separately from answer fluency.
Step 7: Keep the index fresh
Schedule recrawls according to source volatility and the consequence of stale answers. Store the last successful hash per canonical URL. If a new successful crawl has the same hash, skip re-embedding. If it differs, replace the old chunks. Treat a timeout or temporary 403 as unknown, not as proof the source was deleted. Confirm a removal with repeated checks or an authoritative sitemap change before deleting indexed content.
A cron job can launch a Python ingestion command, but the job should write a run record: run ID, start/end times, discovered URLs, accepted pages, changed hashes, deleted pages, failed URLs, and embedding count. For an overview of what Crawl handles versus application-owned work, see Nstdata’s web-data stack guide.
Costs, scaling, and common failure modes
A page count is not a complete cost estimate. Include Crawl requests, separately billed proxy traffic when used, embedding tokens for changed content, vector writes/storage, and answer-time model usage. Your own formula should be:
Measure accepted pages rather than submitted URLs. Common failures include indexing empty pages, mismatching embedding dimensions, leaving stale vectors after an update, losing source URLs during chunking, and measuring only answer quality while ignoring retrieval misses. Add a validation gate before indexing and an adversarial source test before making crawl results available to a model.
Complete ingestion checklist
Keep an allowlist and a URL inventory with canonical URLs.
Validate status, body completeness, title, and source metadata after every crawl.
Preserve headings, code, tables, and content hashes through normalization.
Use deterministic chunk IDs and retain a page-to-chunk mapping.
Re-embed only changed content; distinguish deleted pages from failed requests.
Evaluate retrieval, citations, stale-answer behavior, and cost on a fixed question set.
Fresh web data reaches a RAG answer only after your pipeline discovers, retrieves, validates, normalizes, chunks, embeds, and indexes it. Nstdata Crawl can supply the page retrieval and Markdown conversion layer; the application must still own provenance, update policy, vector lifecycle, and answer checks. Start with a small set of representative URLs, prove coverage and removal behavior, and then scale the schedule.
RAG data ingestion converts source documents into validated, chunked, embedded records that a retrieval system can search.
Q: How do I keep web data in RAG fresh?
Track canonical URLs and content hashes, recrawl on a schedule that matches source volatility, and replace or delete old chunks only after validating the new source state.
Q: Can RAG use live websites directly?
A RAG index does not automatically refresh from a live website; a collector and ingestion job must retrieve and index new content before retrieval can use it.
Q: How often should I update a RAG knowledge base?
Choose a refresh interval from how often the source changes and the impact of a stale answer, then measure actual change rates and update delays.
Q: Does Crawl replace chunking and embeddings?
No. Crawl retrieves and transforms pages; your application still chooses chunks, embedding models, vector storage, update rules, and evaluation.
Understand seven dimensions of RAG data quality and how to validate crawled pages, compare clean and raw inputs, preserve provenance, and diagnose retrieval failures. It separates extraction, retrieval, and generation errors so teams can debug the right stage.
Lena Zhou
Oct. 20th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
