Search discovers URLs; crawling reads pages; RAG retrieves previously indexed content. An agent often needs more than one of these patterns.
Choose on-demand crawling for a known URL that must be current, and scheduled indexing for many repeated questions over the same corpus.
Return source URL, retrieval time, content status, and errors from every web tool call. A page that loads but lacks the target content is a failure for the task.
Treat web content as untrusted data, bound tool calls, and evaluate citation quality as well as retrieval success.
An AI agent asked for today's API limit or a changed product price cannot rely on model memory. It needs a way to obtain a relevant source, read it, and decide whether the observed content supports an answer. The design question is when to search, crawl, cache, or retrieve from an index. Nstdata Crawl fits the page access step when the agent has a public URL to read.
Model training data is a snapshot. Product docs, prices, availability, and policies can change after that snapshot, and a cached knowledge base can be stale too. A web tool can narrow the gap, but “live access” is not a guarantee of truth: a page might be old, personalized, incomplete, or temporarily unavailable. The agent must carry provenance and uncertainty forward instead of converting every page into an authoritative answer.
Consider “What is the current concurrency limit for this API?” A search result might identify the docs URL, a crawl call might retrieve the limit table, and a second source such as a changelog might establish when it changed. Each component has a different job.
Five Common Web Data Access Patterns for AI Agents
Pattern
Trigger
Freshness
Infrastructure
Best fit
Search-first
Unknown source URL
At query time
Search + page reader
Open research
Direct URL crawling
Known URL
At query time
Crawl tool
Product page, current docs
RAG retrieval
Repeated questions
As fresh as index
Crawl, chunk, embed, store
Internal knowledge base
Scheduled refresh
Known changing corpus
At refresh interval
Scheduler + crawl + diff
Catalog or docs updates
Multi-source verification
Consequential claim
At query time or cached
Discovery + multiple readers
Research and fact checks
Pattern 1: Search-First Data Access
Start with a search API when the user gives a topic rather than a source. Filter results by domain, document type, and recency; then crawl the promising URLs. A SERP snippet is a locator, not a complete source. It can omit caveats, numbers, and surrounding context. Limit the number of URLs and reject results outside your allowed domains before invoking the page reader.
Pattern 2: Direct URL Crawling
When the user supplies a URL, a direct crawl avoids a discovery round trip. It works well for a release note, public pricing page, or documentation URL. Validate the URL, request readable content, and return both the content and a retrieval timestamp. Add an explicit content check: a successful response containing only “enable JavaScript” is not useful evidence.
Pattern 3: RAG-Based Retrieval
For repeated questions over a stable set of pages, pre-crawl, clean, chunk, embed, and index the corpus. At query time, retrieve the most relevant chunks, then answer with citations. RAG lowers repeated page fetches but introduces index freshness and chunk-boundary issues. Preserve canonical URLs and a last_crawled_at field in every chunk. Nstdata's URL-to-Markdown guide discusses the upstream collection step.
Pattern 4: Scheduled Data Refresh
Use a scheduler when the same catalog or docs change often enough to justify regular collection. Store a content hash per canonical URL, re-index changed pages, and expire removed pages. A scheduled refresh provides predictable load but cannot prove that data changed only at run time. For sensitive facts, make a new query-time read before the agent acts.
Pattern 5: Multi-Source Research and Verification
Read several independent sources for a claim that matters. Normalize fields and timestamps, then report agreements, disagreements, and unknowns. A single vendor page is usually the best source for that vendor's API syntax, whereas a broader market claim needs more than one source. Do not silently average conflicting prices or technical limits.
How to Choose the Right Data Access Pattern
Ask five questions in order. Is the URL known? If no, search or consult a sitemap. How fresh must the fact be? Query-time read for rapidly changing information; indexed retrieval for frequently reused stable content. How many pages? A few pages suit direct calls; a corpus calls for bounded site crawl and indexing. Does the page require interaction? If the content appears only after login or a complex workflow, a simple crawler may not fit. Will a wrong answer cause an external action? Add source verification and human approval before any high-impact operation.
For an agent that answers questions about a documentation site, the practical architecture is often RAG plus a live crawl fallback: use indexed chunks for most questions, then fetch the canonical page when the index is old or the answer depends on a current limit.
Common Web Data Pitfalls in AI Agents
Stale Data and Missing Context
A retrieved chunk without version, source URL, or collection time is hard to trust. Store last_crawled_at, canonical URL, content hash, and, where available, document version. Define a freshness threshold per topic. A product specification might tolerate a week; a flash sale probably will not. “Fetched today” also does not prove the source itself is current.
Search Results vs Full Webpage Content
Search returns pointers and snippets. A crawl tool returns page content. A vector index returns an older processed excerpt. Using one as a substitute for the others can create false confidence. If the answer depends on a table footnote or a code example, retrieve the full relevant page and inspect the section that contains it.
Prompt Injection and Untrusted Web Content
A webpage can include text such as “ignore previous instructions” or a disguised tool command. Never promote that text into the system or developer instruction channel. Treat it as quoted data, restrict tool permissions, cap URLs and calls, and require user approval before purchases, account changes, or messages to others. See the OWASP guidance on LLM application risks for the threat model.
Other routine failures deserve explicit handling: redirects to unexpected hosts, repeated navigation content, context overflow, 429 responses, and “success” pages with no actual article text. Return a typed error rather than an empty string that the agent may misinterpret as evidence.
Using Nstdata Crawl as an Agent Web Data Source
Nstdata Crawl offers a single-page scrape API and a site crawl API with Markdown and HTML output. Its official documentation also describes JavaScript rendering, browser waits, synchronous and asynchronous tasks, status queries, and Python, Node.js, and Go SDKs. These capabilities reduce retrieval plumbing for public pages; they do not supply search ranking, agent planning, or claim verification. For a broader architecture discussion, see Nstdata's web scraping for AI agents guide.
The useful integration boundary is a small tool returning a stable record. Markdown keeps many headings and links readable for an agent. HTML remains available when extraction needs a page-specific parser. Rendering and waits help with dynamic content, subject to page-specific validation. A high-volume indexing job can use asynchronous crawl tasks; an interactive agent with one known URL can start with synchronous scraping.
Python Integration Example
The following adapter demonstrates a known-URL tool. It requires a valid NSTDATA_API_TOKEN and a locally installed nstdata-ai-crawl package. Replace allowed.example with a domain you are authorized to read. This is a bounded retrieval function, not a full agent implementation.
import os
from datetime import datetime, timezone
from urllib.parse import urlsplit
from nstdata_ai_crawl import Format, NstDataClient, ScrapeRequestDto
ALLOWED_HOSTS ={"allowed.example"}MAX_CONTENT_CHARS =20_000deffetch_web_content(url:str)->dict: parts = urlsplit(url)if parts.scheme !="https"or parts.hostname notin ALLOWED_HOSTS:return{"source_url": url,"status":"rejected","error":"URL not allowed"}try:with NstDataClient(os.environ["NSTDATA_API_TOKEN"])as client: result = client.submit_scrape_task_sync( ScrapeRequestDto( url=url, formats=[Format.MARKDOWN], onlyMainContent=True, timeout=60000,)) markdown = result.data.get_markdown()ifnot markdown ornot markdown.strip():return{"source_url": url,"status":"empty","error":"No usable content"}return{"source_url": url,"retrieved_at": datetime.now(timezone.utc).isoformat(),"status":"ok","content": markdown[:MAX_CONTENT_CHARS],"truncated":len(markdown)> MAX_CONTENT_CHARS,}except Exception as exc:return{"source_url": url,"status":"error","error":str(exc)}
Keep the host allowlist in application configuration, and recheck the final destination if your HTTP layer can follow redirects. Prevent internal-network URLs at the network boundary too; URL string checks alone are insufficient for a production SSRF defense. Pass only the returned content as source data to the model, never as a higher-priority instruction.
How to Evaluate Web Data Quality in Agent Workflows
Use a labeled set of real questions and source pages. Compute source validity as relevant URLs / selected URLs, body completeness as pages containing required sections / fetched pages, freshness as the time between last verified page update and agent answer, citation accuracy as supported cited claims / checked cited claims, and task failure rate as failed end-to-end answers / attempts. Record cost per successfully answered question, including search, crawl, proxy traffic where applicable, model tokens, and retries.
One useful test pairs a deliberately stale index with a changed page. The agent should either refresh the page or label its answer as potentially outdated. Another test embeds an instruction in the page body; the agent should summarize the page without executing that instruction.
They call tools for URL discovery, page retrieval, or indexed search and then reason over the returned evidence. The model itself is not a web crawler.
Should an agent crawl pages or use RAG?
Use crawling for current page reads and corpus refresh. Use RAG for repeated questions over an indexed corpus. Many applications combine both.
What is the best data source for an AI agent?
The source that directly answers the task with sufficient freshness and provenance. Official documentation is usually strongest for API behavior; market research may require several sources.
How can an agent resist prompt injection from a webpage?
Keep page text in a data channel, restrict available tools and domains, limit calls, and require approval before consequential external actions.
Kai Watanabe
Oct. 22nd 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
