For a practical collection layer, review Nstdata Crawl alongside the workflow below.
The original RAG paper describes generation with an external retrieval memory. Retrieval-augmented generation can cite stale material when its index describes an earlier version of a page. A pricing page may change while the old chunks remain searchable; a help article may disappear while its vectors keep answering questions. The model may be behaving consistently with the data it was given, so the correction belongs in the ingestion pipeline.
A freshness target needs two clocks. Collection age is the time since the page was last checked successfully. Index age is the time since the corresponding chunks were written to the retrieval store. They can diverge when extraction succeeds but embedding or indexing fails. Track both. For background on transforming pages into readable text, see URL-to-Markdown conversion and structured extraction from websites.
How web data freshness works in a RAG pipeline
The pipeline is source discovery β page retrieval β main-content Markdown β normalization β change detection β chunking β embedding β vector upsert β retrieval evaluation. Source discovery might come from a sitemap, an approved URL list, or a bounded site crawl. Each stage needs its own error state and retry policy.
Public source URLs -> Nstdata Crawl -> Markdown -> content hash
| unchanged: keep index
| changed: chunk -> embed -> upsert
| removed: verify -> delete vectors
vector store -> retriever -> answer with citations
Nstdata Crawl covers retrieval and extraction. Its current Crawl documentation shows Markdown output through its Python SDK. It does not create embeddings, operate a vector database, decide whether a page changed, or judge whether an answer is current. Those stages belong to your application or other services. For a separate AI workflow pattern, see the CrewAI system guide.
How to detect meaningful page changes
Hash the main text after stable normalization. Ignore incidental blank-line changes, but preserve headings, links, tables, and values that might matter to a user. Do not strip prices, dates, or version labels as βnoise.β A site-wide footer or cookie banner should ideally be removed by the extraction stage; a changing price in the body must produce a new digest.
The hash identifies a content version, not a semantic judgment. A tiny change could matter greatly; a large boilerplate change might not. Review false positives before adding aggressive normalization rules. Store the URL, digest, successful fetch time, extractor version, and index version together. For deletions, require a confirmed permanent response or repeated checks; a transient network error should keep the last valid index version while marking it stale.
Build a website-to-RAG update pipeline
Begin with a small allowlist of public documentation URLs and a clear recrawl cadence. Call the SDK for each URL, calculate the digest, and compare it with your state table. If unchanged, update only last_checked_at. If changed, create chunks with stable IDs such as sha256(url + chunk_index + content_version), calculate embeddings, then upsert them into your chosen vector store. Remove the old version only after the new version is searchable. This order avoids an empty retrieval window.
The snippet below is the acquisition boundary. Install nstdata-ai-crawl and set NSTDATA_API_TOKEN first. The SDK call follows the current Nstdata Python SDK example; a live request still requires your own credential.
import os
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto, Format
defread_markdown(url:str)->str: token = os.environ["NSTDATA_API_TOKEN"]with NstDataClient(token)as client: result = client.submit_scrape_task_sync( ScrapeRequestDto( url=url, formats=[Format.MARKDOWN], timeout=60000, onlyMainContent=True,)) markdown = result.data.get_markdown()ifnot markdown ornot markdown.strip():raise ValueError(f"No readable content returned for {url}")return markdown
Your application then performs the database operations, using the methods and IDs of the chosen store:
defrefresh_one(url, state_store, vector_store, embedder): markdown = read_markdown(url)# Nstdata retrieval; requires an API token digest = content_hash(markdown) old = state_store.get(url)if old and old.digest == digest: state_store.mark_checked(url)return"unchanged" chunks = split_markdown(markdown)# Your chunking policy records =[{"id":f"{url}#{digest}#{i}","url": url,"text": chunk,"vector": embedder.embed(chunk)}for i, chunk inenumerate(chunks)] vector_store.upsert(records)# Third-party vector database state_store.commit(url, digest,[r["id"]for r in records])if old: vector_store.delete(old.vector_ids)return"updated"
split_markdown, state_store, vector_store, and embedder are application interfaces, not Nstdata SDK features. Make the upsert and state commit idempotent: rerunning a failed job should converge on one current version. Follow the vector database's own transaction and deletion semantics; for example, see Pinecone upsert documentation. A production implementation also logs partial writes and retries safely.
Where Nstdata Crawl fits
RAG teams need a reliable web-content input before they can discuss embedding quality. Nstdata Crawl supplies page retrieval, extraction, and Markdown output through an API and SDK; a bounded site crawl can help with URL acquisition when the source permits it. Markdown gives downstream chunkers headings and readable text, while the source URL remains available for citations. The service should be evaluated on the actual sites you are permitted to collect, because page layout and JavaScript behavior vary.
Three practical checks matter. Output quality: verify the returned text contains the expected heading and no login or consent page. Scope: bound discovery and fetch frequency to the source's rules and your budget. Cost: check the current Crawl product and Crawl pricing for the applicable billing model, including separately billed proxy use where relevant. Nstdata provides the input; freshness policy and vector writes remain yours.
Best practices for sustained freshness
Set intervals by source volatility and business impact rather than running every URL on one timer. A product changelog might merit a frequent check; evergreen legal definitions may need fewer. Use conditional scheduling on observed change frequency, but keep a maximum age so quiet sources still get checked. Preserve previous versions for audit and rollback.
Monitor fetch success, extraction quality, hash-change rate, embedding failures, index lag, and citation age. Run representative queries after updates; an index can be technically current while retrieval still returns an irrelevant chunk. Treat personally identifying or licensed material according to source permissions and retention rules. A page being public does not itself grant unlimited reuse.
The useful result is a traceable answer path: question β retrieved chunk β source URL β content version β last successful check. That makes a stale answer debuggable instead of mysterious.
Update frequency depends on how fast the source changes and the cost of a stale answer. Set a maximum allowed age per source, then measure actual change and index lag.
Q: How do you detect changes in web pages?
Compare a hash of normalized main content with the last successful version. Keep fetch failures separate from content changes.
Q: Can Markdown be used for RAG?
Yes. Markdown preserves useful structure for chunking, but it still needs validation, embeddings, indexing, and retrieval tests.
Q: How do you update a vector database when a page changes?
Write new versioned chunks, verify they are searchable, then delete the prior chunk IDs. Use the chosen database's documented upsert and delete operations.
Lena Zhou
Oct. 27th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more β without managing crawling infrastructure.
