One Platform, Three Products: The Nstdata Approach
TL;DR
Nstdata’s three foundational web-data products are Proxy, Crawl, and Proxy Manager; the broader platform also includes Cloud Browser.
Nstdata Crawl is a Web Crawling API for LLM-ready data with site discovery, bounded recursive crawl, JavaScript rendering, multiple outputs, and built-in proxy orchestration.
Nstdata Proxy supplies direct network access when an application controls its own requests; Proxy Manager adds pools, routing, logs, and monitoring to proxy workflows.
Use Crawl for website-to-content work, Proxy for custom clients, and Proxy Manager when route-level operations need shared control.
A unified vendor does not remove application-level validation, source policy, or data storage.
What Does “One Platform, Three Products” Mean?
“One platform, three products” describes a practical Nstdata web-data architecture: Proxy supplies network access, Crawl acquires and converts site content, and Proxy Manager organizes routing and proxy operations. This is a way to understand the three foundational layers, not a claim that Nstdata sells only three products. The public platform also presents Cloud Browser for browser-oriented automation.
The Nstdata platform presents these products alongside Cloud Browser. The products solve related but different engineering problems. A team with an existing scraper might only need direct proxy connectivity. A RAG team might need a managed crawl and page output. An operations team managing multiple proxy workflows might need pooling, routing rules, and observability. Buying all three is not a prerequisite for using any one of them.
What Does Nstdata Crawl Do for AI Workflows?
Nstdata Crawl is a Web Crawling API for LLM-ready data that turns approved pages or a bounded site into usable page artifacts. It can discover pages from a seed, follow links recursively under maxDepth and maxPages limits, render JavaScript when needed, and return Markdown, HTML/raw content, links, PDF, or screenshots according to the request. The and describe these capabilities.
Crawl orchestrates proxies internally for a standard job. You do not enter a separate proxy Host, Port, Username, Password, and Protocol before using the Crawl API. That reduces setup and makes the crawl request the main integration point. It does not imply free proxy consumption, guaranteed access, or automatically correct business fields. A page may render successfully yet still contain duplicate navigation, a challenge, or an ambiguous product variant.
For RAG, store source URL, final URL, fetch time, content hash, and accepted Markdown with each document. For structured extraction, validate the field values against the page. For audit or troubleshooting, retain an HTML or screenshot artifact under a deliberate retention policy. The RAG knowledge-base article covers the next stage after acquisition.
When Should a Developer Use Nstdata Proxy?
A developer should use Nstdata Proxy when their application already owns the HTTP client, browser, parser, and crawl scheduler but needs a proxy network for permitted requests. The application configures the protocol and endpoint credentials in its own client. This gives fine-grained request control and keeps application-specific extraction in existing code.
That freedom comes with work: discover URLs, control concurrency, choose retry behavior, handle rendering if required, parse output, and monitor failures. A proxy is a transport component; it does not convert a website into a corpus. If the task is to collect an approved documentation site as Markdown, Crawl provides a shorter route. If a custom client requires precise request headers or a domain-specific parser, direct Proxy may fit better.
When Does Proxy Manager Add Value?
Proxy Manager adds value when several proxy-using workflows need a shared place to define pools and routes and inspect traffic behavior. Its documentation describes Router, pool, routing, logs, and monitoring concepts. A team can use it to keep routing decisions visible instead of scattering proxy configuration across services.
Proxy Manager is not a content extractor or vector database. It does not decide which page is relevant, whether a response is a challenge, or how to map a product page to a normalized record. Use it for network operations when direct proxy workflows justify that layer. A team using only managed Crawl may not need to set it up.
How Do You Connect the Products Without Overbuilding?
Connect the product that matches the first missing capability, then add another only when a measured requirement appears. For a RAG pilot, obtain a Crawl API credential, submit an approved seed with bounded depth and page count, poll task status, validate returned documents, and index only accepted content. For a pre-existing scraper, configure a Proxy endpoint in the HTTP client and record status plus content validation separately. For a fleet of such clients, add Proxy Manager when routing rules and logs need centralized ownership.
A minimal application-owned Crawl record could be:
This is an illustrative downstream schema, not a claimed API response. Keep the API’s task ID and page results in an adapter, and avoid embedding vendor-specific response fields throughout the application. Recheck the current API reference before implementing endpoints or payloads. The web data pipeline guide describes the downstream normalization and storage stages.
The platform boundary is clear: Nstdata can help acquire pages, render them, return artifacts, and manage proxy traffic, while your application remains responsible for lawful source selection, schema accuracy, retention, and downstream AI behavior. A single vendor may simplify procurement and integration; it does not guarantee that every public site will be accessible or that every output will satisfy your quality bar.
Review Crawl pricing together with proxy usage, rendering needs, and the number of accepted documents. Track discovered, attempted, fetched, and accepted pages as separate counts. Calculate cost per accepted document. If the job is a custom scraper, compare direct Proxy costs plus your own operational time. For context on the wider stack, see web data infrastructure for AI teams.
Which Nstdata Product Should You Start With?
Start with Crawl when the target is a bounded website corpus or page-level content for an AI product. Start with Proxy when the application already has a crawler and needs network access. Add Proxy Manager when multiple proxy workflows require shared routing and diagnostics. Consider Cloud Browser for browser session automation rather than treating it as one of these three layers.
The next action is a small pilot with an accepted-output definition and an explicit cost ceiling. If the source changes often, define a refresh and deletion policy before scaling; the data-freshness guide explains that operational requirement.
