Introduction: RAG goes stale when the index stops moving
Nstdata Crawl is the collection layer in this workflow; its product overview describes page and site retrieval for downstream data pipelines.
A RAG system may query its vector database on every request and still return old information. The source page may have changed, but the index may not have been recrawled. Or new vectors may exist while a retrieval cache still contains old results. “Fresh” therefore describes more than the model’s answer time.
This guide designs a scheduled update pipeline for public documentation and knowledge pages: inventory → crawl → validate → compare → chunk/embed → upsert/delete → verify. It assumes your app controls the source allowlist and vector database. Nstdata Crawl handles retrieval and returns task/page results; it does not decide whether a document is obsolete for your product. See the Crawl API documentation for current task and result endpoints.
What RAG freshness actually means
Freshness layer
Question
Example timestamp
Source freshness
When did the page itself last change?
source_updated_at if available
Index freshness
When did your accepted content reach the vector store?
last_indexed_at
Answer freshness
Which source snapshot informed this answer?
retrieved crawled_at
A crawl timestamp does not prove a page was updated, and an index timestamp does not prove the latest source was retrieved. Keep the timestamps that your system can actually observe; do not invent source_updated_at if the site does not expose it.
Architecture of an automatically updated knowledge base
A production job has a scheduler, a durable task record, a crawl worker, a normalizer, a content-hash comparison, an embedding stage, a vector-store writer, and a validation check. Each step needs an idempotency key and a visible result. A run should be safe to repeat after a worker crash.
For any page, store at least canonical_url, last_attempt_at, last_success_at, content_hash, version, status, and the current chunk IDs. Keep historical crawl artifacts separately if auditability matters. The manifest is the control plane that tells the pipeline which records to replace.
Step 1: Build a URL inventory
Start with an explicit allowlist of public pages or a bounded discovery process. A sitemap helps enumerate likely URLs; links help find pages omitted from it. Normalize fragments and tracking parameters, but do not collapse query strings if they change page content. Versioned docs should remain distinct (/v1/ and /v2/) unless the application has a clear rule for retiring one.
Persist the inventory before a large run. If a new crawl is interrupted, you can resume from the last completed URL instead of rediscovering what the job already did.
Step 2: Schedule recrawling at the right interval
The refresh schedule should reflect source volatility and business risk. Pricing or API behavior may warrant a shorter interval than an archived company history page. Start with a schedule, observe the percent of pages that change, and adjust. Avoid “recrawl everything every minute” as a default: it increases requests without necessarily improving answers.
For simple deployments, a cron entry can invoke one idempotent job:
A managed scheduler or queue is preferable when jobs need retries, concurrency limits, per-site pacing, or durable checkpoints. Nstdata’s current docs show asynchronous Crawl task IDs and status queries; they state that webhooks are not currently supported. Have the worker poll status with a deadline and backoff rather than waiting forever for an external callback.
Full recrawling versus incremental recrawling
A full recrawl can simplify discovery when the site is small and bounded. Incremental processing reduces embedding and vector writes when most pages are unchanged, but it requires a URL manifest and reliable hash comparison. Many systems combine both: rediscover URLs on a slower schedule, then recrawl known high-priority pages more frequently.
Step 3: Detect changed, new, and deleted pages
Hash normalized content, not the raw HTML bytes. Raw markup can change because of request IDs, timestamps, or layout while the useful text stays the same. Keep the canonical URL and normalized body with the hash so you can reproduce the comparison.
from hashlib import sha256
import re
defnormalize_for_hash(markdown:str)->str: text = markdown.replace("\r\n","\n") text = re.sub(r"\n{3,}","\n\n", text)return text.strip()defcontent_hash(markdown:str)->str:return sha256(normalize_for_hash(markdown).encode("utf-8")).hexdigest()
If the hash matches the last successful record, skip embedding. If it differs, build a replacement chunk set and verify it before retiring old IDs. If the page is new, create its initial record. If a URL disappears from a sitemap, flag it as a deletion candidate; sitemap omission alone is not enough to remove a document.
Handle removed pages and redirects safely
Treat response conditions differently:
Successful page: validate content, compare hash, then update if changed.
404 or 410: record the status and confirm according to your site policy before deleting.
301 or 302: retain the original URL mapping and evaluate the canonical destination; do not blindly duplicate both versions.
403, 429, timeout, or network error: mark the attempt as failed/unknown and keep the last known good index.
A page can be temporarily inaccessible. Use a confirmation window or a second independent check before deleting it. When a redirect consolidates two pages, update citations and references deliberately.
Step 4: Rechunk and re-embed only the new version
Rechunk when the normalized text changes or the chunking algorithm changes. Re-embed when the text changes or the embedding model changes. Record embedding_model, chunker_version, and the page content hash with the index generation so a rollback can identify which records are compatible.
The OpenAI embeddings guide describes the input-to-vector API. Keep a queue or retry table for transient embedding failures so one failed batch does not cause all successfully processed URLs to repeat.
Step 5: Replace vectors without leaving duplicates
Use deterministic chunk IDs plus a page manifest. With Chroma, upsert replaces existing IDs and delete removes records; its current collection API reference documents those operations. With Pinecone, keep per-page IDs or a page prefix and use its upsert and delete guidance. For either system, do not assume an upsert removes old chunks whose IDs are no longer in the new page version.
A safe replacement sequence is:
Write the candidate generation under new IDs or a versioned namespace.
Query the candidate records and confirm expected sources and passages are retrievable.
Switch the application’s active generation or replace the manifest pointer.
Delete previous IDs after the new generation passes validation.
If your vector store cannot provide a convenient version switch, keep the old IDs in a manifest and replace them only after the full upsert succeeds. The trade-off is a short overlap window, which you can handle by filtering on active version metadata.
Step 6: Validate freshness with known changes
Use a handful of pages that you can safely edit or that change predictably. Record a query whose correct answer depends on the new wording. After ingestion, verify that the latest chunk is retrievable and that the old chunk is absent from the active namespace. Also test one page that is unavailable to confirm that the pipeline does not delete its previous good record.
Track at least:
Metric
Definition
Crawl-to-index delay
last_indexed_at - last_successful_crawl_at
Change detection rate
changed pages / successfully crawled pages
Failed-page rate
failed page attempts / submitted page attempts
Stale chunk rate
known old versions still active / sampled changed pages
Citation age
answer time minus retrieved record’s crawl time
These are operational metrics, not a guarantee of answer quality. Pair them with a labelled retrieval and citation test set.
Step 7: Monitoring, recovery, and costs
Write one run record with run_id, discovered count, attempted count, successful count, changed count, deleted count, embedding tokens, vector writes, failed URLs, and elapsed time. Redact tokens and any sensitive page content from logs. Use bounded retries with exponential backoff for transient errors, and dead-letter persistent failures for review. Add a run lock so two schedulers do not replace the same page simultaneously.
For a 10,000-page corpus with a hypothetical 5% daily change rate, the change set is about 500 pages before page-level failures and rediscovery effects. Estimate crawl cost for the pages you actually request, then embedding cost for the changed text, vector writes/storage, proxy traffic if used, and worker/scheduler operations. This is a model, not a quoted price. Check the latest Nstdata Crawl pricing and your embedding/vector provider’s billing docs before budgeting.
Event-driven versus scheduled updates
A CMS, Git repository, or documentation platform may emit an event when content changes. Use that event to enqueue a URL only if the source system can reliably identify the page and the event is authorized. A periodic reconciliation job is still useful for missed events and deletions. An event trigger in your application does not mean the Crawl API itself supplies a webhook; current Crawl docs direct users to query task status.
Common update failures
The most damaging errors are deleting a page after a timeout, hashing unstable navigation instead of normalized content, generating new IDs on every run, changing the embedding model without a new index generation, and forgetting to remove old chunks after edits. A cached answer can also look stale even after the index is current; invalidate retrieval/application caches or include the index generation in the cache key.
Complete automated update checklist
Store a canonical URL inventory and page-to-chunk manifest.
Separate successful, failed, redirected, and deleted states.
Hash normalized content and skip unchanged embeddings.
Keep the last known good version until new vectors pass validation.
Test known changes, deletions, redirects, and temporary failures.
Monitor crawl-to-index delay, failure rate, stale versions, citations, and costs.
Keeping a RAG knowledge base current is a data lifecycle problem. A scheduler starts the work, but freshness depends on durable identity, careful change detection, safe vector replacement, and validation against real questions. Treat failures as unknown, not deletion, and record the timestamps needed to explain what the system knew when it answered.
Q: How do I update a RAG knowledge base automatically?
Schedule a URL inventory job, crawl and validate pages, compare normalized content hashes, re-embed changed pages, and safely replace old vector records.
Q: How often should RAG data refresh?
Set the interval from each source’s change rate and the risk of stale answers, then adjust it using observed changes and crawl-to-index delay.
Q: What is incremental RAG indexing?
Incremental indexing processes only new, changed, or removed source records instead of rebuilding the entire index on every run.
Q: How do I remove outdated embeddings?
Keep a page-to-chunk ID manifest, validate the replacement records, and then delete the old IDs or retire the previous index generation.
Q: Can RAG update automatically?
Yes, when your application runs a scheduled or event-triggered ingestion job that retrieves, validates, and indexes source changes.
Understand seven dimensions of RAG data quality and how to validate crawled pages, compare clean and raw inputs, preserve provenance, and diagnose retrieval failures. It separates extraction, retrieval, and generation errors so teams can debug the right stage.
Lena Zhou
Oct. 20th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
