Spark Structured Streaming With Web Data: Concepts, Use Cases, and Production Considerations
TL;DR
Spark Structured Streaming processes new records incrementally; it does not discover or render web pages by itself.
A reliable web-data architecture runs a bounded crawler upstream, lands immutable page events in a supported source, and lets Spark consume those events.
Nstdata Crawl is a Web Crawling API for LLM-ready data: it can discover pages within explicit limits, render JavaScript, and return Markdown, HTML, raw data, links, PDFs, or screenshots.
Treat a successful crawl submission as a job acceptance, not proof that every page succeeded; reconcile completed, failed, pending, and total counts.
Use a stable page key, observation time, checkpoint, and idempotent sink so restarts do not silently double count updates.
Evaluate cost per accepted, timely record rather than requests alone.
What does Spark Structured Streaming with web data actually mean?
Spark Structured Streaming with web data means continuously processing newly collected page records through Spark's incremental DataFrame engine. It is useful when a product team needs changing prices, public policy pages, documentation, or market signals to reach a warehouse or AI index without rebuilding an entire batch dataset each time. Apache Spark's Structured Streaming overview describes the micro-batch execution model and checkpoint-based recovery.
For page collection, Nstdata Crawl provides the upstream Web Crawling API for LLM-ready data.
The distinction matters: a URL is not a streaming source. Spark can read files, Kafka, and other supported sources, while a crawler handles discovery, access, rendering, and extraction. Calling an HTTP API inside foreachBatch is a side effect of a streaming sink callback; it does not turn that API into a supported source with source offsets. The automated data collection guide shows the broader collection-to-consumption pattern.
Experience Nstproxy Crawl - Start Your Free Trial Today
When is this architecture worth using?
This architecture is worth using when many independently changing pages feed the same downstream transformations and freshness matters more than a daily full export. A retailer may watch permitted product pages for offer changes; an AI team may refresh a RAG corpus when public documentation changes; a risk team may track public policy revisions. The business value comes from smaller, traceable updates and faster decisions, provided the source observations are actually fresh.
For a small set of URLs refreshed once a day, a scheduled batch job is usually simpler. Structured Streaming adds checkpoint storage, schema management, sink semantics, and continuous operations. Choose it only if those costs buy a useful freshness or throughput gain. The automated data collection overview helps set a service-level target before selecting a processing mode.
What components make a production web-data stream?
A production web-data stream has a bounded acquisition job, a durable landing layer, a structured transformation, and an observable destination. The flow is:
Seed URLs or reviewed sitemap → Nstdata Crawl → job reconciliation → immutable JSON events → Kafka or object storage → Spark Structured Streaming → validated table or index
Nstdata Crawl can submit a site crawl from an entry URL with maxDepth, maxPages, includeUrls, excludeUrls, and ignoreQuery. Those controls keep a catalog section from expanding into search pages, account routes, or endless query variants. Its browser rendering can capture content that appears after JavaScript executes, and its outputs serve different consumers: Markdown for RAG, HTML or raw data for field extraction, links for discovery, and screenshots or PDFs for investigation. Built-in proxy orchestration means the caller does not have to configure a separate proxy endpoint. These are collection capabilities, not a promise that a page's business fields are already normalized. See the Nstdata Crawl API documentation for the current request and response contract.
The landing event should include source_url, observed_at, crawl_id, page_status, content_hash, and a normalized content or artifact reference. Keep the raw response or a durable reference for debugging. Avoid putting a large PDF or screenshot into a Kafka message; store it separately and pass a reference. If the API returns a large-artifact reference, retrieve it through the documented storage-read endpoint instead of guessing a URL.
How do you connect Nstdata Crawl to Spark without inventing a connector?
Connect Nstdata Crawl to Spark through a scheduler and a durable source that Spark already supports. The example below uses a bounded site-crawl submission and a JSON-lines landing directory. It is an integration pattern, not a native Nstdata Spark connector.
Step 1: Submit a bounded crawl
Create an Nstdata account, obtain a Crawl API key, choose a permitted public seed URL, and set the scope before starting the job. Store the key in NSTDATA_API_KEY. The documented site-crawl endpoint is POST https://api.nstdata.io/api/v1/crawl with the x-api-key header.
curl-sS-X POST 'https://api.nstdata.io/api/v1/crawl'\-H"x-api-key: $NSTDATA_API_KEY"\-H'Content-Type: application/json'\--data'{"url":"https://docs.example.org/guide/","formats":["markdown","links"],"maxDepth":2,"maxPages":100,"includeUrls":["*docs.example.org/guide/*"],"excludeUrls":["*docs.example.org/guide/search*"],"ignoreQuery":true,"onlyMainContent":true}'
Replace the example host with a URL you are authorized to collect. The response should contain a crawl ID; record it with the scheduled run. Do not assume submission means collection completed. The website-to-Markdown guide covers seed and scope decisions.
Step 2: Reconcile the job and land accepted page events
Poll GET /api/v1/crawl/status/{crawlId} until the task reaches a terminal state, then inspect total, completed, failed, and pending. Retrieve pages through GET /api/v1/crawl/pages/{crawlId}. The current docs show nextCursor in the response but do not clearly define the cursor request parameter, so verify the paging contract with your account or SDK before claiming complete retrieval at scale. Persist one immutable event per accepted page, partitioned by observation date and run ID. Write new files atomically so Spark never sees half-written JSON.
Step 3: Process landed events with a checkpoint
Define a schema rather than asking a stream to infer one from changing pages. A minimal PySpark consumer can read line-delimited JSON files as they arrive:
The code requires PySpark and directories writable by the Spark process. It demonstrates the supported file-source boundary; it cannot run against the Nstdata service without your API key and landing process. For a production sink, add validation, deduplication, and a stable key such as canonical URL plus content hash or observation version. Keep checkpoint state on reliable storage. Spark's streaming DataFrame API guide documents sinks and foreachBatch semantics.
How do you know the stream is correct and fresh?
The stream is correct and fresh only when the crawler and Spark agree on coverage, latency, and accepted records. Compare requested and discovered URLs, completed and failed pages, landed files, Spark input rows, and destination rows. Sample rendered pages against their Markdown or HTML; a 200 response can still contain a login screen, empty shell, or wrong locale. Track observed_at to destination availability, not merely Spark processing latency.
Set explicit acceptance rules: required fields present, nonempty main content, expected URL scope, acceptable language, no duplicate version, and an error budget for failures. Keep dead-letter events with a reason and source URL. For AI retrieval, inspect chunk quality and provenance before indexing. A generic page artifact is not automatically an accurate product or policy fact.
What risks and cost boundaries should you plan for?
The main risks are uncontrolled crawl expansion, stale or incomplete page content, schema drift, repeated processing, and hidden operating cost. Bound discovery with depth, page count, and URL filters; separately cap scheduler concurrency. JavaScript rendering can increase latency, and a changed page layout can break field mapping even when the crawl succeeds. Do not infer a billing rule or a fixed price from a request count; verify your plan and measure cost per accepted page, including rejected and retried work.
Also confirm that collection complies with site terms, applicable law, privacy obligations, and internal policy. Avoid authentication bypass and non-public pages. If a strict event-level SLA or low-latency stream is required, test source-to-sink latency against representative sites before committing to the architecture.
Which approach should your team choose?
Choose scheduled batch for a small, slow-changing URL set, and choose Structured Streaming when a durable page-event feed and continuous downstream processing justify the added operations. Start with one bounded Nstdata Crawl job, reconcile every page, and land immutable events; then connect the event store to Spark. The same Markdown output can also feed an RAG web-data workflow once quality checks are in place.
Q: Can Spark Structured Streaming crawl a website directly?
Spark Structured Streaming does not provide website discovery, browser rendering, or crawl-state management. Use a crawler upstream and feed Spark through a supported durable source.
Q: Is a successful Nstdata Crawl submission enough to create a Spark event?
A successful submission only confirms that the job was accepted. Wait for page results, inspect status and content, then emit accepted events.
Q: Should every page update become a new row?
Every observation can be an event, but downstream tables should apply a documented version or deduplication rule. Keep source_url, observation time, and content hash so repeated crawls are explainable.
Q: Does Nstdata Crawl replace Spark transformations?
Nstdata Crawl supplies accessed and processed web artifacts. Spark or another downstream system still handles domain-specific schema mapping, joins, quality rules, and delivery.
Kai Watanabe
Nov. 3rd 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
