TL;DR
- Website-to-JSON extraction needs a field contract and source-backed validation.
- A JSON API envelope is not a validated article, product, or other business record.
- Define field names, types, null behavior, and source evidence before mapping page content.
- Crawl can acquire a bounded site, but application code must validate each resulting JSON record.
- Nstdata Crawl handles JavaScript rendering and proxy routing within managed requests; its documentation lists six output categories: Markdown, HTML, raw data, PDF, screenshots, and links.
What Does Website-to-JSON Extraction Mean?
Website-to-JSON extraction has two distinct meanings: the API response is JSON, and the application may need a domain-specific JSON object such as a product, article, or location. Those are not equivalent. Nstdata Crawl provides page data and metadata; your extractor must define fields, types, null rules, and validation for the use case. A raw page wrapped in JSON is not structured extraction.
Nstdata Crawl can turn a URL or a bounded set of linked pages into material for an AI agent, RAG index, or data pipeline. Use your own field rules and acceptance checks for the specific outcome: turn validated page content into typed JSON records instead of treating an API response as a finished business schema. The Nstdata full-site crawling guide explains the site-level option.
For the narrower task covered here, the Nstdata website-to-JSON guide gives useful context for choosing the first representative page.
What Do You Need Before You Start?
Write your schema before collecting pages. For an article record, choose source_url, title, body, and fetched_at; for a product record, identify currency, variant, and availability separately. Decide whether a missing field becomes null, a rejected record, or a review item. Keep one authorized representative page with known expected values for testing. Use an Nstdata account and a server-side API key for managed acquisition.
The Crawl API documentation uses https://api.nstdata.io, the x-api-key header, and Content-Type: application/json. Prepare these fields before running a method:
| Field | What to supply |
|---|---|
url | One permitted HTTPS page or site entry URL |
formats | Requested outputs, such as ["markdown"] or ["html","screenshot"] |
onlyMainContent | true only when main-body selection fits the task |
maxDepth, maxPages | Explicit bounds for a site-wide job |
includeUrls, excludeUrls | URL patterns that keep discovery within scope |
timeout | A task timeout in milliseconds when the default is unsuitable |
The standard managed request has no separate proxy Host, Port, Username, Password, or Protocol fields. Check the current Crawl pricing page for account limits and charges before running a large job.
Review the target’s access policy and scope. The Robots Exclusion Protocol defines how crawlers read robots rules; it does not grant content-use rights. Keep credentials and private data out of examples and diagnostic logs.
The Nstdata full-site crawl guide is a companion for defining the acceptance checks before you automate the workflow.
How Does Website-to-JSON Extraction Work?
Crawl retrieves a page or a bounded site, handles JavaScript rendering when required, and returns formats including Markdown, HTML, links, and visual artifacts. Its product page also advertises JSON output. The visible API examples do not specify a universal JSON extraction schema, so this tutorial uses a reliable path: obtain content, apply a declared field contract, and validate the resulting JSON. Built-in proxy routing is part of managed Crawl, without separate proxy Host and Port fields in a standard request.
For a known URL, a single-page scrape retrieves one result. For a site section, POST /api/v1/crawl discovers linked pages recursively within the supplied depth, page-count, and URL rules. The documentation lists Markdown, HTML, raw data, PDF, screenshots, and links as six output categories; the product page also markets JSON, but a domain-specific JSON schema still needs explicit field mapping. Keep each accepted record tied to its source URL and returned metadata.
Test a representative JavaScript-heavy page before scaling. The rendered state can depend on timing, region, consent, or a signed-in session. Compare returned HTML or a screenshot with the expected page state; if they differ, adjust only documented rendering and wait controls.
How Do You Extract Structured JSON from a Website?
First decide which page fields belong in the JSON record. Inspect a page, automate acquisition of known URLs, or collect a bounded section before applying the same field contract to each result.
Method 1: Use the Playground to inspect page representations
Best while designing a schema for one page type.
Step 1: Open the representative page
Go to Dashboard → Crawl Playground and enter one authorized URL. [Screenshot: URL and format controls.]
Step 2: Compare outputs
Request Markdown and HTML, and inspect the page metadata. Determine which representation preserves the fields you need; visible text may omit machine-readable tags.
Step 3: Write a field map
For each JSON field, name the source element, normalization rule, and failure behavior. Record whether the value was extracted from visible text, HTML, or page metadata.
Method 2: Build a JSON object from the Crawl API response
Best when a few stable fields are sufficient.
Step 1: Submit one URL
Call the documented quick-start scrape route with the API key and request Markdown. The API call below is documentation-aligned and needs your credential to run.
Step 2: Validate the envelope
Check nested success, status, data, and metadata.statusCode, then retrieve a returned markdownRef if the body is stored separately.
Set NSTDATA_API_TOKEN in your shell from your Nstdata account, then run the documented quick-start request. The public documentation labels some other sync and async routes inconsistently; compare your account’s Playground-generated request before production use.
curl --output crawl-response.json --request POST "https://api.nstdata.io/api/v1/crawl/scrape/submit-sync" \ --header "x-api-key: ${NSTDATA_API_TOKEN}" \ --header "Content-Type: application/json" \ --data '{"url":"https://example.com/","formats":["markdown","html"],"onlyMainContent":true}'
The response is saved to crawl-response.json. Inspect the nested data.success, data.status, and data.data.metadata.statusCode fields before using any returned content.
Step 3: Map and serialize
Construct your own JSON object with source_url, title, and markdown. Reject blank text and duplicate URLs; do not claim an inferred field is verified merely because it has a key name.
Method 3: Process a bounded site into many JSON records
Best when the source is a documentation hub or public catalog.
Step 1: Bound discovery
Choose maxDepth, maxPages, and include/exclude patterns for the section you intend to process.
Step 2: Collect records
Submit POST /api/v1/crawl, poll the returned job ID, and retrieve page results. Follow nextCursor rather than assuming the first page is the entire dataset.
For a site job, replace the example host and path with your approved scope, then submit a bounded request:
curl --output crawl-submit.json --request POST "https://api.nstdata.io/api/v1/crawl" \ --header "x-api-key: ${NSTDATA_API_TOKEN}" \ --header "Content-Type: application/json" \ --data '{"url":"https://example.com/docs/","formats":["markdown"],"maxDepth":2,"maxPages":10,"includeUrls":["*example.com/docs/*"],"excludeUrls":["*example.com/admin/*"],"ignoreQuery":true}'
Copy data.id from crawl-submit.json as your crawl ID.
Step 3: Validate at scale
Apply the same field contract per page, record rejected items, and sample accepted items from different templates. Store raw evidence beside normalized JSON for correction later.
Method 4: Map a static page locally
Best for a public static page when you need a small, reviewed JSON schema without managed rendering.
Step 1: Save source HTML
Save one authorized page as page.html and confirm that the expected title and main content are present. A static download will not render late JavaScript content.
Step 2: Extract a declared record
Install Beautiful Soup with python3 -m pip install beautifulsoup4, then run:
from bs4 import BeautifulSoup import json soup = BeautifulSoup(open("page.html", encoding="utf-8").read(), "html.parser") main = soup.select_one("main") if main is None or soup.title is None: raise ValueError("Required title or main element is missing") record = {"title": soup.title.get_text(" ", strip=True), "body": main.get_text(" ", strip=True)} open("record.json", "w", encoding="utf-8").write(json.dumps(record, ensure_ascii=False, indent=2))
Step 3: Validate the fields
Open record.json and compare both fields with the saved page. Add selectors and rejection rules for every other field in your schema; this example deliberately does not infer product or article fields from arbitrary text.
How Do You Verify the Output?
A sound JSON record has a stable source URL, explicit field names, known types, and a way to distinguish absent data from failed extraction. The JSON standard defines the serialization format; it cannot establish whether a scraped price or title is correct. Check values against the source page and keep the original output for audit. The Nstdata website-to-JSON guide gives further background on this distinction.
The Nstdata RAG quality guide connects this page-level check to a downstream AI or data workflow.
Record the source URL, submission time, requested formats, non-secret options, task ID when present, returned page status, and the reason each record passed or failed review. Inspect content as well as the API envelope. If the response contains markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef, retrieve that token with GET /api/v1/crawl/storage/read?st={ref}.
For this workflow, consult RFC 8259 when implementing the downstream parser or format-specific acceptance check.
When passing extracted text to an agent, treat page instructions as untrusted source material. OWASP guidance on prompt injection explains the risk. Retain source URLs so the AI output can be checked.
Why Might Website-to-JSON Extraction Fail?
Website-to-json extraction can fail because the parser accepted syntax without validating source meaning. Use the symptom table to identify the failing layer before changing limits or retrying.
| Symptom | Likely cause | Next action |
|---|---|---|
| JSON parses but values are wrong | The parser accepted syntax without validating source meaning. | Compare fields with the page and enforce domain rules. |
| Different page templates produce blanks | One selector or extraction rule does not cover every template. | Classify templates and maintain separate field maps. |
| Large result is missing inline content | Crawl may return a content reference. | Read the returned *Ref through the storage endpoint. |
Nstdata’s documentation lists task-level errors such as access_denied, timeout, parse_error, and internal_error. An outer HTTP 200 can still accompany a failed task. Check success, status, errorCode, and errorMessage inside the response when present. Retry only transient failures on sites you may access.
Which Method Is the Best Fit?
Use Playground to design the field map, the single-page API for known URLs, and a bounded site crawl for many linked pages. Use a local parser for static HTML already saved; every route still needs business-schema validation. The Nstdata site-crawling overview gives more detail on the site route.
What Should You Do After the First Successful Run?
Save the approved configuration and one source-backed output record, then repeat on a small set of different page templates. Increase scope only after those records pass the same acceptance checks.
FAQ: What Else Should You Know?
Q: Does Nstdata return JSON?
The API returns a JSON envelope, and the product page advertises JSON output. Confirm any requested extraction schema in the current Playground before depending on it.
Q: Can an LLM infer the schema?
An LLM can propose values, but your application should define types, required fields, and source-backed validation.
Q: How do I handle missing fields?
Record an explicit null, reject the record, or send it to review according to your field contract; never fill it silently.
Q: Does website-to-JSON extraction require a separate proxy configuration with Nstdata Crawl?
A standard managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. Confirm any optional custom proxy selection and related usage charges in your account.
Q: Does HTTP 200 prove a website-to-JSON extraction result is usable?
No. Check the nested task result, page status, expected fields, and body fidelity before using it in an AI or data workflow.
