Agentic Search: Architecture, Retrieval, and Tutorial
TL;DR
Agentic search is an iterative loop in which a model plans queries, discovers sources, retrieves full evidence, evaluates gaps, and searches again until a stopping rule is met.
Search-result snippets are discovery signals, not sufficient evidence; a reliable system needs a deep page retrieval layer that returns complete, attributable content.
Nstdata Crawl can serve as that deep retrieval layer by turning selected URLs or bounded sites into page artifacts before model reasoning.
Production quality depends on source diversity, freshness, retrieval acceptance checks, prompt-injection controls, citations, and explicit time, token, and request budgets.
Start with a narrow research question and a small retrieval budget, then measure citation correctness and answer completeness before increasing autonomy.
What Is Agentic Search?
Agentic search is a search workflow where an AI system decides what to look for, evaluates what it found, and performs follow-up retrieval based on unresolved questions. Unlike a single query-and-answer pipeline, it forms a feedback loop: plan, search, fetch, extract evidence, reason, identify gaps, and repeat.
The word “agentic” should describe control flow, not marketing. A system is meaningfully agentic when intermediate evidence changes the next query or retrieval action. If an application sends one query to a search API and summarizes the top snippets, it is search-augmented generation, but it is not a deep iterative search agent.
In the architecture below, Nstdata Crawl is the deep page retrieval layer between URL discovery and evidence-based reasoning.
Why Agentic Search Matters in 2026
Agentic search matters because many useful questions cannot be answered from one ranking page or one model context. Vendor evaluations need product documentation, status pages, security records, and user experience. Investment research needs filings, investor-relations pages, current news, and market data. Technical investigation may require documentation, release notes, source code, issues, and reproducible tests.
Recent research describes deep search as an integration of autonomous reasoning, iterative retrieval, and synthesis rather than a single lookup. The is useful for understanding the feedback-loop framing. Firecrawl's similarly separates retrieval, orchestration, and reasoning. Production systems, however, need simpler controls than research prototypes: bounded tools, auditable evidence, and clear stopping criteria.
A practical agentic search system has four separable layers.
Layer
Responsibility
Typical output
Main failure
Orchestration
Plan steps, budgets, retries, and stopping
Search and fetch actions
Loops or premature stopping
Discovery
Find candidate URLs and source types
Ranked URLs and snippets
Ranking bias or missing sources
Deep retrieval
Fetch and normalize full pages or documents
Markdown, HTML, metadata, artifacts
Blocks, empty shells, wrong locale
Reasoning
Compare evidence and compose answer
Claims, uncertainty, citations
Hallucination or source misuse
Discovery Is Not Deep Retrieval
Discovery services optimize for finding promising URLs. They often return title, URL, and a short snippet. That is enough to decide what to fetch, but not enough to support detailed claims. Snippets can be truncated, stale, or detached from page context.
Deep retrieval opens the selected page, executes required rendering, extracts the main content, and returns enough metadata to verify what was read. Keeping this as a distinct tool lets the orchestrator retry a failed page, substitute another source, or refresh evidence without repeating the whole search.
Reasoning Must Be Evidence-Bounded
The reasoning layer should receive a bounded set of sources with explicit provenance. It should distinguish source statements from inference and identify conflicts instead of blending them into one confident sentence.
A good answer includes fewer claims with strong evidence rather than many claims supported only by topical similarity. Citation presence is not citation correctness: evaluators must verify that the cited passage actually entails the claim.
Nstdata Crawl as the Deep Page Retrieval Layer
Nstdata Crawl fits the deep retrieval position between URL discovery and model reasoning. The application supplies a selected public URL or a bounded site task; Crawl handles retrieval and returns documented page artifacts. The agent then reasons over accepted output rather than search snippets.
This separation is useful because retrieval failures differ from reasoning failures. A page can return a consent screen, wrong locale, incomplete JavaScript shell, or soft block. The retrieval adapter can reject those results before they enter the model context.
Nstdata Crawl also supports workflows beyond plain text. Depending on the current endpoint and format, a pipeline may request Markdown, HTML, links, screenshots, or PDFs. Visual artifacts help when meaning depends on layout, while links can support bounded follow-up discovery. Confirm formats and request fields in the live documentation or SDK before implementation.
Tutorial: Build an Agentic Search Pipeline With Nstdata Crawl
The following architecture uses any search provider for discovery, Nstdata Crawl for deep page retrieval, and an LLM for planning and synthesis. It intentionally avoids binding the system to one model or search API.
Step 1: Define the Research Contract
Write the question, required source types, freshness window, geographic scope, and completion criteria before running the agent. For example:
{"question":"What changed in Vendor X's API during the last 90 days?","required_sources":["official documentation","official changelog","official status or incident page"],"max_search_rounds":3,"max_pages":12,"freshness_days":120}
A contract prevents the agent from interpreting “more search” as unlimited search. It also gives evaluation a concrete standard.
Step 2: Generate Source-Oriented Queries
The planner should produce queries for missing source types rather than synonyms of the original question. Example queries might include the vendor domain plus “API changelog,” “breaking changes,” or “incident.” Use domain filters when authoritative first-party evidence is required.
Store each query and why it was issued. If the agent cannot explain which evidence gap a query addresses, do not spend the request.
Step 3: Deduplicate and Prioritize URLs
Normalize scheme, hostname casing, fragments, trailing slashes, and known tracking parameters. Prefer canonical first-party pages for product facts. Keep independent primary research or credible user discussions when they answer a different question, such as operational experience.
Do not retrieve every search result. Score candidates by authority, relevance, freshness, source-type coverage, and duplication. A shortlist of diverse sources is usually better than ten pages repeating the same announcement.
Pin a tested version in production and keep the API credential in a secret manager.
Step 5: Implement the Deep Retrieval Adapter
import os
from nstdata_ai_crawl import NstDataClient, ScrapeRequestDto, Format
client = NstDataClient(api_key=os.environ["NSTDATA_API_KEY"])defretrieve_page(url:str): request = ScrapeRequestDto( url=url, formats=[Format.MARKDOWN],)return client.scrape(request)
This adapter follows the public SDK's documented types but requires a credential, so it is a prerequisite-gap example. Inspect the actual response object in your environment and map documented task states rather than assuming field names.
Step 6: Add Retrieval Acceptance Checks
A successful transport response is not enough. Validate:
the final host is expected;
the page language matches the research scope;
a canonical title or required marker is present;
content exceeds a task-specific minimum;
the result is not a login page, consent wall, or soft block;
retrieval time and source URL are stored;
duplicate content hashes are collapsed.
Rejected pages should produce structured failure reasons. The orchestrator can then retry, change retrieval options, or choose a different source.
Step 7: Extract Evidence, Not Just Summaries
For every accepted page, ask the model or a deterministic parser to produce claim-sized evidence objects.
{"claim":"The API removed parameter X.","source_url":"https://vendor.example/changelog","retrieved_at":"2026-08-31T00:00:00Z","evidence":"Short supporting passage","confidence":"high","source_type":"official changelog"}
Keep excerpts short and within copyright limits. Store enough surrounding context or offsets internally to audit the claim later.
Support agent retrieval with managed proxy routing
Use Nstdata proxy infrastructure when agent workflows need controlled, authorized network access.
After each round, the reasoner should output answered subquestions, unresolved subquestions, conflicting evidence, and missing source types. The planner may issue another query only for a recorded gap.
A useful stopping rule ends when required source types are covered and every material claim has support, or when the round, page, token, or time budget is exhausted. “The model feels done” is not an operational rule.
Step 9: Compose With Claim-Level Citations
Generate the final answer from accepted evidence objects, not raw search results. Attach citations immediately after supported claims. State uncertainty when sources conflict or when only secondary evidence is available.
Run a citation audit that checks three things: the URL resolves, the cited content contains the evidence, and the evidence supports the exact claim. A polished answer with decorative citations still fails this gate.
When to Crawl a Site Instead of Individual Pages
Use bounded site crawling when the relevant documents are distributed across a known domain and discovery through search is incomplete. Documentation portals, changelog archives, and investor-relations sites are common examples.
Set explicit page limits, depth, include patterns, exclude patterns, and query-parameter handling. Start with a map or shallow crawl when available. Unbounded crawling can enter calendars, faceted navigation, localization duplicates, or session URLs and waste the research budget.
For open-web questions, individual URL retrieval usually works better. Search providers discover candidates across domains; Nstdata Crawl then fetches only the high-value pages.
Security, Compliance, and Prompt Injection
Agentic search expands the attack surface because external pages influence subsequent model actions. Treat all retrieved content as untrusted data. A page may tell the agent to ignore instructions, disclose secrets, or call another tool. The orchestrator must prevent page text from changing system policy.
Use tool allowlists, request budgets, destination controls, private-network blocking, credential isolation, and human approval for high-impact actions. Do not send secrets in URLs or page forms. Respect website terms, robots expectations where applicable, copyright, and personal-data obligations.
Research involving financial, health, employment, or personal data needs stronger review. Retrieval capability does not authorize collection or automated decisions.
How to Evaluate Agentic Search
Evaluate the system on answer quality and evidence quality, not how many tool calls it makes. Useful metrics include:
claim correctness and completeness;
citation entailment and source quality;
coverage of required source types;
retrieval acceptance rate;
duplicate-page rate;
freshness compliance;
latency and cost per accepted answer;
number of unnecessary search rounds;
rate of successful prompt-injection resistance tests.
Create a stable evaluation set with questions that require multiple sources, changing facts, and at least one retrieval failure. Re-run it when the model, search provider, crawler configuration, or prompt changes.
Common Failure Modes
The first failure is snippet synthesis: the agent answers from result summaries without opening pages. The second is source monoculture, where several results repeat one press release. The third is runaway exploration without a stopping budget.
Other common failures include retrieving the wrong locale, treating a status code as content success, and attaching a citation to a related but non-supporting page. Most are pipeline problems, not model-intelligence problems.
Agentic search is an iterative evidence workflow, not a search box with a longer answer. Reliable systems separate discovery, deep retrieval, reasoning, and evaluation so each failure can be observed and corrected.
Start with one narrow research contract, no more than a few search rounds, and a small page budget. Use Nstdata Crawl as the deep retrieval layer when you need managed page or bounded-site collection, then expand autonomy only after citation correctness and retrieval acceptance meet your target.
Q: What is the difference between agentic search and RAG?
Agentic search dynamically plans and repeats retrieval based on evidence gaps, while a basic RAG system usually retrieves from a predefined index once per question. Agentic search can feed a RAG store or query one as a source.
Q: Does agentic search need a web crawler?
Agentic search needs full-content retrieval, but not every question requires site crawling. A crawler is useful for bounded multi-page domains; individual page retrieval is better for selected open-web URLs.
Q: Can Nstdata Crawl replace a search API?
No. Nstdata Crawl is positioned here as the deep page retrieval layer after discovery. A search API finds candidate URLs, while Crawl retrieves selected pages or bounded sites.
Q: How many search rounds should an agent use?
Use the fewest rounds that cover required source types and resolve material gaps. Set a hard maximum based on risk, latency, and cost; three rounds is a reasonable starting experiment, not a universal rule.
Q: How do I prevent hallucinations in agentic search?
Require claim-level evidence, validate retrieval output, preserve provenance, audit citation entailment, and state uncertainty. These controls reduce hallucinations but do not guarantee that every source or model conclusion is correct.
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
