Data Freshness: Why Real-Time Web Data Beats Snapshots
TL;DR
Web data freshness is the age of the information your product serves, not simply the timestamp of its last crawl.
An hourly fetch does not guarantee an hourly fresh answer; extraction, validation, indexing, cache, and retrieval each add lag.
Define source-specific freshness targets. Product price, availability, documentation, and archival pages do not need the same refresh interval.
Prefer change-aware re-crawls and versioned records over blindly replacing the entire corpus on a timer.
Nstdata Crawl can reacquire and convert selected web pages through its API; your application must schedule refreshes and measure end-to-end lag.
What Does Data Freshness Mean for a Web Data Product?
Data freshness means that a user sees information sufficiently close to the source's current state for the decision being made. A timestamp on a fetched page only proves when acquisition occurred. A fresh acquisition can still be stale to the user if the parser extracted an old cached element, the new record is waiting in a queue, or the retrieval index still serves a previous version. The Nstdata Crawl product is a managed option for the acquisition and conversion steps discussed here.
Define three timestamps for a record: source_observed_at when the page was fetched, processed_at when a validated record was produced, and served_at or index version when the product made it available. The source's actual change time is often unknown. When a source exposes a reliable Last-Modified header or change feed, record it, but do not assume it reflects every content change.
The end-to-end path contains multiple delays: scheduling, fetch, rendering, parsing, validation, queueing, indexing, cache expiry, and answer generation. A product that fetches every minute but indexes once daily has roughly daily freshness. A product that indexes immediately but fails to notice changed URLs has the same problem.
Measure served_at - source_observed_at as pipeline lag and now - source_observed_at as age at read time. Neither alone proves source currency, because the page might have changed after the fetch. For high-risk use cases, compare a sample against the live source and estimate detection lag. Track the 95th percentile by source class rather than an average across all pages.
Start with the cost of stale data. A changed product price or stock status can affect a transaction within minutes. A developer documentation correction may matter within hours. A glossary definition might tolerate weeks. Set an explicit service objective, such as “95% of approved product records are observed and indexed within 30 minutes of a detected change,” only if your source and pipeline can actually support it.
Source class
Typical decision
Design choice
Price or inventory
Is this offer current?
Short refresh interval, fast validation, visible fetched time
Documentation
Does this API still work?
Change-aware recrawl, versioned index, rollback
News or events
What happened recently?
Frequent discovery and de-duplication
Reference pages
What is the stable definition?
Longer interval with event-triggered update
The intervals above are design categories, not universal settings. Review each source's terms, allowed access pattern, and technical limits before choosing a rate.
How Do You Build a Refresh Loop That Produces Fresh Answers?
Method 1: Scheduled Re-crawl for a Small Site Set
Best for: A known list of approved sites whose update frequency is predictable.
Step 1. Store each source's URL, owner, refresh target, last successful fetch, content hash, and next due time.
Step 2. Schedule a bounded recrawl. For Nstdata Crawl, submit the documented site or single-page API request with the relevant discovery limits and retain its job ID.
Step 3. Poll job status, fetch completed page output, validate the content, then atomically replace the serving version. Keep the previous good version on failure.
Limit: A fixed interval can waste requests on quiet pages and miss changes between runs.
Method 2: Change-Aware Re-crawl for Large Corpora
Best for: Many URLs with uneven change rates.
Step 1. Prioritize URLs by measured change frequency and business impact. Give new or frequently changing pages shorter revisit intervals.
Step 2. Compare normalized content hashes or selected fields after fetching. An unchanged HTML wrapper can still contain changed data, while a changed advertisement can make the whole-page hash noisy.
Step 3. Reprocess and re-index only changed accepted records, then record the old and new versions. Adjust intervals using observed change rates and error signals.
Limit: Conditional HTTP headers such as ETag and Last-Modified help only when the origin implements them reliably and the fetching path exposes them. Verify this on the actual source.
Method 3: Event-Driven Updates When a Source Provides Them
Best for: First-party or partner sources with feeds, webhooks, or change events.
Step 1. Subscribe to the source's documented change signal and authenticate it according to that source's instructions.
Step 2. Queue the changed URL or record ID for fetch and validation; treat duplicate events as normal.
Step 3. Reconcile periodically against the source inventory so missed events do not create permanent gaps.
Limit: A third-party website usually does not provide your product with a webhook. Do not describe polling as real-time event delivery.
Nstdata Crawl can discover pages, recursively crawl within configured limits, render JavaScript, and return Markdown, HTML, raw data, links, screenshots, and PDF through the Crawl API. Built-in proxy routing means the application does not need a separate proxy configuration for standard managed collection. This simplifies repeat acquisition from approved sites, especially when some pages need browser rendering.
The documentation describes job submission, status polling, and page retrieval. The scheduler, change detector, freshness objective, index update, and user-visible timestamp belong to your application. A failed Crawl job should be recorded as a freshness failure rather than silently retaining an old record under a current date.
How Do You Verify That Users Receive Current Data?
Test the full serving path. Select a sample of source pages with known recent changes, compare the live page to the stored record, verify that the current index points to the newest accepted version, and ask the product a question whose answer changed. For RAG, check that the cited source URL and version match the answer. For analytics, check that stale and rejected records are excluded from current-state calculations.
Publish a visible last_verified_at when age matters to the user's decision. If the freshness target is breached, mark the record stale or suppress it; do not present an old price or policy as live. The Azure RAG design guide treats data preparation and evaluation as part of solution quality, which is why refresh needs to be checked beyond the crawler.
How Do You Balance Freshness, Cost, and Source Load?
Treat refresh allocation as a scheduling problem. For each page or source class, estimate change frequency, impact of a stale value, fetch cost, rendering cost, and permitted request budget. Prioritize pages whose stale values cause the most harm. Revisit low-change pages less often, but reserve a small audit budget to detect when their behavior changes. A change-aware policy can reduce wasted fetches without claiming perfect knowledge of the next source update.
Use a freshness budget across stages. If the product promises an hour from observed change to served record, allocate time for detection, acquisition, validation, indexing, and cache invalidation. A queue that routinely consumes 50 minutes of that hour leaves little room for rendering failures or retries. Publish stage-specific lag metrics so the owning team can see where time is lost.
Do not make the source carry unnecessary load. Honor documented APIs or feeds when available, use conditional requests when supported, and back off on 429 or 503. Increasing request frequency without verifying the source's limits can reduce reliability for both parties. Built-in proxy routing helps operationalize permitted collection, but it does not justify exceeding a site's access policy.
What Happens When a Source Change Is Partially Processed?
Promote a new version only after all required fields and artifacts are validated. A page may be fetched at 10:00, parsed at 10:02, indexed at 10:10, and served at 10:20. Keep those timestamps rather than overwriting one updated_at field. If the 10:00 fetch is a consent screen, it should be rejected while the last validated version remains available with its original observation time.
For multi-page entities, such as a product listing plus detail page, define whether the product requires an atomic update. A new price from one page paired with an old availability from another can create an impossible record. Either publish fields with their individual observation times or wait for a complete entity refresh. The correct choice depends on the decision the user makes from that data.
Finally, test cache behavior. A search index may hold the new record while an application cache serves the old answer. A RAG answer may cite a new document but use a previously cached summary. Include cache invalidation and answer-level checks in the freshness objective, not just crawler status.
Q: Is real-time web data the same as a one-minute crawler interval?
No. Real-time describes the product's useful latency from source change to served record, and every pipeline stage contributes to that latency.
Q: Should every page be re-crawled at the same rate?
No. Set refresh frequency by source change rate, decision impact, and permitted access pattern.
Q: What should happen when a refresh fails?
Keep the last validated version, mark its age accurately, retry within a bounded policy, and alert when the freshness objective is breached.
This Python tutorial moves from one static page to validated records and managed rendering. It includes runnable examples, failure diagnosis, source controls, and an AI-ready data contract.
Marcus Chen
Oct. 29th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
