Getting Structured Product Data from E-commerce Pages

TL;DR

• Product data extraction succeeds only when price, currency, variant, and availability refer to the same offer.
• A product record is valid only when price, currency, availability, and variant refer to the same offer.
• Product JSON-LD is a useful candidate source, but compare it with the rendered selected offer.
• Use bounded discovery for a reviewed catalog section and keep category pages out of product records.
• Nstdata Crawl handles JavaScript rendering and proxy routing within managed requests; its documentation lists six output categories: Markdown, HTML, raw data, PDF, screenshots, and links.

What Counts as Structured Product Data?

Structured product data is a typed record for a specific product or variant: name, identifier, selected option, price, currency, availability, and source URL. E-commerce pages often show several offers, regional prices, subscriptions, bundles, or selected variants on one screen. A generic page-to-JSON wrapper does not resolve which value belongs to which offer. Define the record identity and evidence for each field before scaling.

Nstdata Crawl can turn a URL or a bounded set of linked pages into material for an AI agent, RAG index, or data pipeline. Use your own field rules and acceptance checks for the specific outcome: retrieve permitted product pages and map visible or embedded details into validated records. The Nstdata full-site crawling guide explains the site-level option.

For the narrower task covered here, the Nstdata website-to-JSON guide gives useful context for choosing the first representative page.

Collect product-page evidence Use Nstdata Crawl to retrieve permitted product pages and map visible or embedded details into validated records. Explore Crawl

https://example.com/article Crawl M↵Markdown {}JSON {   "title": "...",   "url": "..." } Screenshot

What Do You Need Before You Start?

Select a public or authorized product page and record the selected variant, expected SKU, price text, currency, availability, and last-updated context. Determine whether the site offers an official feed or API; use that where appropriate. Set rules for sale versus list price, out-of-stock labels, multiple sellers, and regional currency. Keep sensitive customer or account data out of the workflow.

The Crawl API documentation uses https://api.nstdata.io, the x-api-key header, and Content-Type: application/json. Prepare these fields before running a method:

The standard managed request has no separate proxy Host, Port, Username, Password, or Protocol fields. Check the current Crawl pricing page for account limits and charges before running a large job.

Review the target’s access policy and scope. The Robots Exclusion Protocol defines how crawlers read robots rules; it does not grant content-use rights. Keep credentials and private data out of examples and diagnostic logs.

The Nstdata e-commerce scraping guide is a companion for defining the acceptance checks before you automate the workflow.

How Do Product Pages Expose Structured Data?

Nstdata Crawl can render JavaScript-heavy product pages, acquire Markdown or HTML, and crawl a bounded category section. Its product page advertises JSON outputs, while the visible API example does not specify an arbitrary product schema. Use Crawl as the retrieval and cleaning layer, then parse published Product structured data or reviewed DOM elements and validate against the visible page. Managed proxy routing is built in; the standard request does not require separate proxy credentials.

For a known URL, a single-page scrape retrieves one result. For a site section, POST /api/v1/crawl discovers linked pages recursively within the supplied depth, page-count, and URL rules. The documentation lists Markdown, HTML, raw data, PDF, screenshots, and links as six output categories; the product page also markets JSON, but a domain-specific JSON schema still needs explicit field mapping. Keep each accepted record tied to its source URL and returned metadata.

Test a representative JavaScript-heavy page before scaling. The rendered state can depend on timing, region, consent, or a signed-in session. Compare returned HTML or a screenshot with the expected page state; if they differ, adjust only documented rendering and wait controls.

How Do You Extract Product Data from E-commerce Pages?

Compare one rendered offer with its embedded markup, then automate known product URLs or a bounded catalog section. Map fields only after the selected variant and region are clear.

Method 1: Inspect one product in the Playground

Best for establishing what the page actually exposes.

Step 1: Open a representative product

Use Dashboard → Crawl Playground with an authorized product URL. [Screenshot: URL, rendering, and output formats.]

Step 2: Request HTML and visual evidence

Choose HTML and screenshot outputs. If price or stock loads after JavaScript, use a documented readiness wait visible in the Playground.

Step 3: Compare variants

Check the selected variant in the screenshot against any embedded Product JSON-LD and visible price. Do not mix a parent product name with a different variant’s offer.

Method 2: Retrieve the page through the API

Best for repeatable collection from known product URLs.

Step 1: Store the key securely

Keep NSTDATA_API_TOKEN on the server and send it as x-api-key.

Step 2: Request the page

Use the documented quick-start scrape shape below for HTML and Markdown. Check the current Playground-generated route because the docs label other scrape routes inconsistently.

Set NSTDATA_API_TOKEN in your shell from your Nstdata account, then run the documented quick-start request. The public documentation labels some other sync and async routes inconsistently; compare your account’s Playground-generated request before production use.

curl --output crawl-response.json --request POST "https://api.nstdata.io/api/v1/crawl/scrape/submit-sync" \
  --header "x-api-key: ${NSTDATA_API_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{"url":"https://example.com/product/sample","formats":["html","markdown"]}'

The response is saved to crawl-response.json. Inspect the nested data.success, data.status, and data.data.metadata.statusCode fields before using any returned content.

Step 3: Map fields

Parse embedded Product markup when available and compare its values with visible content. If the cleaned HTML omits JSON-LD scripts, inspect a raw-data representation in your account before assuming there is no structured markup.

If the page publishes Product JSON-LD, save raw HTML as page.html and inspect its declared offers before mapping fields. The following code extracts Product nodes without pretending every store uses the same offer schema:

from bs4 import BeautifulSoup
import json

soup = BeautifulSoup(open("page.html", encoding="utf-8").read(), "html.parser")
nodes = []
for script in soup.select('script[type="application/ld+json"]'):
    try:
        data = json.loads(script.string or script.get_text())
    except json.JSONDecodeError:
        continue
    stack = data if isinstance(data, list) else [data]
    while stack:
        item = stack.pop()
        if isinstance(item, dict):
            if item.get("@type") == "Product" or "Product" in (item.get("@type") or []):
                nodes.append(item)
            graph = item.get("@graph", [])
            stack.extend(graph if isinstance(graph, list) else [graph])
if not nodes:
    raise ValueError("No Product JSON-LD; inspect the rendered DOM")
for product in nodes:
    print(json.dumps({"name": product.get("name"), "sku": product.get("sku"),
                      "offers": product.get("offers")}, ensure_ascii=False, indent=2))

Install Beautiful Soup with python3 -m pip install beautifulsoup4. Treat each printed offer as a candidate: compare variant, price, currency, and availability with the rendered page before storing a product record.

Method 3: Crawl a bounded catalog section

Best for a reviewed category or a short set of related products.

Step 1: Set scope

Start from a category URL and set a small maxDepth, maxPages, and include/exclude rules to avoid cart, search, and account paths.

Step 2: Collect page records

Submit POST /api/v1/crawl, poll status, and retrieve all pages. Identify product detail pages explicitly; category pages are not product records.

For a site job, replace the example host and path with your approved scope, then submit a bounded request:

curl --output crawl-submit.json --request POST "https://api.nstdata.io/api/v1/crawl" \
  --header "x-api-key: ${NSTDATA_API_TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{"url":"https://example.com/docs/","formats":["html"],"maxDepth":2,"maxPages":10,"includeUrls":["*example.com/docs/*"],"excludeUrls":["*example.com/admin/*"],"ignoreQuery":true}'

Copy data.id from crawl-submit.json as your crawl ID.

Step 3: Validate offers

Deduplicate by stable product and variant IDs, normalize currency without changing meaning, and reject pages where the selected offer cannot be tied to the extracted values.

Method 4: Use published Product JSON-LD from a static page

Best for public product pages whose original HTML already contains complete Product markup.

Step 1: Save the product HTML

Save one authorized product page as page.html and note the selected variant, displayed price, currency, and stock state.

Step 2: Extract candidate offers

Run the Product JSON-LD parser shown in Method 2. If it finds no Product node, do not fabricate fields; use a rendered representation or an official product feed if one is available.

Step 3: Validate the selected offer

Choose the offer matching the selected variant and compare every stored value with the visible product page. Static markup can be stale or omit region-specific prices; use Crawl's rendered page and screenshot when that distinction matters.

How Do You Verify the Output?

A valid record names the product or variant, keeps the source URL, and ties price, currency, and availability to the same visible offer. Store original price text beside normalized values so a reviewer can catch locale mistakes. Google’s Product structured-data documentation distinguishes product snippets, merchant listings, and variant markup. The Nstdata website-to-JSON guide explains why schema validation alone cannot prove a scraped value is true.

The Nstdata price-monitoring guide connects this page-level check to a downstream AI or data workflow.

Try Nstdata for Free →

Record the source URL, submission time, requested formats, non-secret options, task ID when present, returned page status, and the reason each record passed or failed review. Inspect content as well as the API envelope. If the response contains markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef, retrieve that token with GET /api/v1/crawl/storage/read?st={ref}.

For this workflow, consult Google Product structured data when implementing the downstream parser or format-specific acceptance check.

When passing extracted text to an agent, treat page instructions as untrusted source material. OWASP guidance on prompt injection explains the risk. Retain source URLs so the AI output can be checked.

Why Might Product Records Contain the Wrong Offer?

Product data extraction can fail because the page shows several offers or default variants. Use the symptom table to identify the failing layer before changing limits or retrying.

Nstdata’s documentation lists task-level errors such as access_denied, timeout, parse_error, and internal_error. An outer HTTP 200 can still accompany a failed task. Check success, status, errorCode, and errorMessage inside the response when present. Retry only transient failures on sites you may access.

Which Method Is the Best Fit?

Use Playground to review a variant, the API for known product URLs, or a bounded site crawl for a reviewed catalog section. Use local JSON-LD parsing on a static page only when its offers match the selected variant. The Nstdata site-crawling overview gives more detail on the site route.

What Should You Do After the First Successful Run?

Save the approved configuration and one source-backed output record, then repeat on a small set of different page templates. Increase scope only after those records pass the same acceptance checks.

Try Nstdata for Free →

FAQ: What Else Should You Know?

Q: Does Crawl produce a finished product schema?

The public API documentation confirms page outputs but does not clearly define arbitrary product-field extraction. Map and validate fields in your application.

Q: Can product JSON-LD replace visible checks?

No. Structured markup may be stale or refer to a different variant; compare it with the rendered offer.

Q: Should I store product prices as floating-point numbers?

Prefer decimal-safe values or original text plus currency; avoid silent rounding and locale assumptions.

Q: Does product data extraction require a separate proxy configuration with Nstdata Crawl?

A standard managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. Confirm any optional custom proxy selection and related usage charges in your account.

Q: Does HTTP 200 prove a product data extraction result is usable?

No. Check the nested task result, page status, expected fields, and body fidelity before using it in an AI or data workflow.