Metadata extraction needs both page-level response metadata and explicit HTML tag parsing.
Open Graph properties are HTML meta tags; a Crawl response title alone does not provide every OG field.
Preserve the document head, repeated og:image values, final URL, and canonical URL in an audit.
Resolve relative image URLs and flag missing or conflicting fields instead of guessing values.
Nstdata Crawl handles JavaScript rendering and proxy routing within managed requests; its documentation lists six output categories: Markdown, HTML, raw data, PDF, screenshots, and links.
What Website Metadata Should You Extract?
Website metadata includes the document title, description, language, canonical URL, and optional social-preview tags such as og:title, og:description, og:image, and og:url. Crawl’s response metadata can describe the fetched page, but Open Graph fields live in HTML <meta> elements and need explicit parsing unless a documented endpoint exposes them. Keep original tag values and the final URL so a later audit can distinguish an absent tag from a parser failure.
Nstdata Crawl can turn a URL or a bounded set of linked pages into material for an AI agent, RAG index, or data pipeline. Use your own field rules and acceptance checks for the specific outcome: collect page titles, descriptions, canonical URLs, and Open Graph fields from a bounded set of pages. The Nstdata full-site crawling guide explains the site-level option.
For the narrower task covered here, the Nstdata site crawl guide gives useful context for choosing the first representative page.
Define fields and duplicate-handling rules before scanning a site. Record whether you need the initial URL, final URL, canonical URL, or og:url; they can differ. Decide how to handle multiple og:image values, relative image URLs, empty descriptions, and localized pages. Use only public or authorized pages, a bounded URL scope, and a server-side Nstdata key for managed collection.
The Crawl API documentation uses https://api.nstdata.io, the x-api-key header, and Content-Type: application/json. Prepare these fields before running a method:
Field
What to supply
url
One permitted HTTPS page or site entry URL
formats
Requested outputs, such as ["markdown"] or ["html","screenshot"]
onlyMainContent
true only when main-body selection fits the task
maxDepth, maxPages
Explicit bounds for a site-wide job
includeUrls, excludeUrls
URL patterns that keep discovery within scope
timeout
A task timeout in milliseconds when the default is unsuitable
The standard managed request has no separate proxy Host, Port, Username, Password, or Protocol fields. Check the current Crawl pricing page for account limits and charges before running a large job.
Review the target’s access policy and scope. The Robots Exclusion Protocol defines how crawlers read robots rules; it does not grant content-use rights. Keep credentials and private data out of examples and diagnostic logs.
Open Graph properties are stored in HTML meta elements such as meta[property="og:title"], usually in the document head. Nstdata Crawl can retrieve single pages or a bounded site, render JavaScript when needed, and return HTML with page metadata. Use the API for acquisition, then parse the document head or returned HTML for Open Graph and canonical elements. The service’s managed proxy routing requires no separate proxy Host, Port, Username, or Password in the standard request. A site crawl can discover many pages, but the application still maps and validates each tag.
For a known URL, a single-page scrape retrieves one result. For a site section, POST /api/v1/crawl discovers linked pages recursively within the supplied depth, page-count, and URL rules. The documentation lists Markdown, HTML, raw data, PDF, screenshots, and links as six output categories; the product page also markets JSON, but a domain-specific JSON schema still needs explicit field mapping. Keep each accepted record tied to its source URL and returned metadata.
Test a representative JavaScript-heavy page before scaling. The rendered state can depend on timing, region, consent, or a signed-in session. Compare returned HTML or a screenshot with the expected page state; if they differ, adjust only documented rendering and wait controls.
How Do You Extract Website Metadata and Open Graph Tags?
Inspect one page head, retrieve the required HTML representation, and scale the same field map across a bounded site. Keep response metadata separate from tags parsed out of the page.
Method 1: Inspect one page in the Playground
Best for confirming whether the required tags survive the chosen HTML output.
Step 1: Open a representative page
Navigate to Dashboard → Crawl Playground and enter an authorized URL. [Screenshot: URL and HTML output controls.]
Step 2: Request HTML and metadata
Choose HTML output and run the page scrape. Check the returned title and language fields, then inspect whether the HTML includes meta[property="og:title"] and a canonical link.
Step 3: Record the field map
Write down the exact tag or response field for each output key. If cleaned HTML removed document-head tags, choose a raw-data mode in the current Playground or another approved source representation.
Method 2: Acquire HTML through the API
Best for repeatable metadata checks on known URLs.
Step 1: Protect the credential
Store NSTDATA_API_TOKEN server-side and send it as x-api-key.
Step 2: Fetch a representative page
Use the documented scrape quick-start request below. The current documentation’s scrape-route labels conflict elsewhere, so verify the generated request from your Playground.
Set NSTDATA_API_TOKEN in your shell from your Nstdata account, then run the documented quick-start request. The public documentation labels some other sync and async routes inconsistently; compare your account’s Playground-generated request before production use.
curl--output crawl-response.json --request POST "https://api.nstdata.io/api/v1/crawl/scrape/submit-sync"\--header"x-api-key: ${NSTDATA_API_TOKEN}"\--header"Content-Type: application/json"\--data'{"url":"https://example.com/","formats":["html"]}'
The response is saved to crawl-response.json. Inspect the nested data.success, data.status, and data.data.metadata.statusCode fields before using any returned content.
Step 3: Parse and validate tags
Read the HTML or returned artifact reference, parse meta[property], meta[name], and link[rel=canonical], then resolve relative URLs against the final page URL. Distinguish missing tags from an empty capture.
Save a representation that includes the document head as page.html; if cleaned HTML omits it, retrieve the returned raw-data reference. This parser keeps repeated Open Graph values instead of overwriting them:
from bs4 import BeautifulSoup
from urllib.parse import urljoin
import json
source_url ="https://example.com/page"# replace with the returned final URLsoup = BeautifulSoup(open("page.html", encoding="utf-8").read(),"html.parser")head = soup.head
if head isNone:raise ValueError("No document head; request raw page data")tags ={}for tag in head.select("meta[property], meta[name]"): key = tag.get("property")or tag.get("name") tags.setdefault(key.lower(),[]).append(tag.get("content",""))canonical = head.select_one('link[rel="canonical"]')record ={"source_url": source_url,"title": soup.title.get_text(" ", strip=True)if soup.title elseNone,"description": tags.get("description",[None])[0],"canonical": urljoin(source_url, canonical.get("href"))if canonical elseNone,"open_graph":{k:[urljoin(source_url, v)if k in{"og:image","og:url"}else v for v in vals]for k, vals in tags.items()if k.startswith("og:")},}print(json.dumps(record, ensure_ascii=False, indent=2))
Install Beautiful Soup with python3 -m pip install beautifulsoup4. Compare title, canonical, og:title, and og:url with the live page; a missing key remains missing rather than being guessed from body text.
Method 3: Audit a bounded site
Best for SEO inventories or social-preview quality checks.
Step 1: Set site boundaries
Choose maxDepth, maxPages, and include/exclude rules for the public section.
Step 2: Collect every page record
Submit a site crawl, poll the job, and retrieve all result pages using the returned cursor.
For a site job, replace the example host and path with your approved scope, then submit a bounded request:
curl--output crawl-submit.json --request POST "https://api.nstdata.io/api/v1/crawl"\--header"x-api-key: ${NSTDATA_API_TOKEN}"\--header"Content-Type: application/json"\--data'{"url":"https://example.com/docs/","formats":["html"],"maxDepth":2,"maxPages":10,"includeUrls":["*example.com/docs/*"],"excludeUrls":["*example.com/admin/*"],"ignoreQuery":true}'
Copy data.id from crawl-submit.json as your crawl ID.
Step 3: Export an audit table
Write one row per canonical page with source URL, HTTP status, title, description, Open Graph fields, and validation flags. Review duplicates and mismatches rather than silently choosing one tag.
Method 4: Inspect metadata on a static page without Crawl
Best for a few public pages when their original HTML already contains the social tags.
Step 1: Download the authorized HTML
Use an approved browser or HTTP client to save the page as page.html. Confirm the saved file contains the document head and expected og: tags.
Step 2: Run the tag parser
Run the Python parser in Method 2 against page.html, setting source_url to the final URL actually downloaded. Keep repeated tag values rather than overwriting them.
Step 3: Compare the live preview fields
Check title, canonical URL, og:title, og:url, and image destinations against the page. Static HTML may differ from a JavaScript-modified document head, so use a rendered capture for such sites.
How Do You Verify the Output?
A good metadata record reflects the actual document head and keeps the exact source and final URLs. Compare og:title with the document title, validate image URL resolution, and flag canonical/OG URL conflicts for review. The Open Graph protocol defines its core properties; do not infer an absent property from page body text. The Nstdata SEO crawling guide helps scope a larger audit.
Record the source URL, submission time, requested formats, non-secret options, task ID when present, returned page status, and the reason each record passed or failed review. Inspect content as well as the API envelope. If the response contains markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef, retrieve that token with GET /api/v1/crawl/storage/read?st={ref}.
For this workflow, consult Open Graph protocol when implementing the downstream parser or format-specific acceptance check.
When passing extracted text to an agent, treat page instructions as untrusted source material. OWASP guidance on prompt injection explains the risk. Retain source URLs so the AI output can be checked.
Why Might Metadata or Open Graph Tags Be Missing?
Website metadata extraction can fail because cleaned HTML may omit the document head. Use the symptom table to identify the failing layer before changing limits or retrying.
Symptom
Likely cause
Next action
OG tags are missing
Cleaned HTML may omit the document head.
Inspect raw page data or another source representation before declaring tags absent.
Image URLs do not open
The page uses relative URLs or malformed values.
Resolve against the final URL and test the destination.
Many pages share one title
Templates, redirects, or a consent page may have been captured.
Compare status, final URL, and screenshot before accepting records.
Nstdata’s documentation lists task-level errors such as access_denied, timeout, parse_error, and internal_error. An outer HTTP 200 can still accompany a failed task. Check success, status, errorCode, and errorMessage inside the response when present. Retry only transient failures on sites you may access.
Which Method Is the Best Fit?
Use Playground to confirm that the document head is present, the API for known pages, or a bounded site crawl for many linked pages. Parse static HTML locally when rendering and managed routing are unnecessary. The Nstdata site-crawling overview gives more detail on the site route.
What Should You Do After the First Successful Run?
Save the approved configuration and one source-backed output record, then repeat on a small set of different page templates. Increase scope only after those records pass the same acceptance checks.
Q: Does the Crawl response include Open Graph tags automatically?
The visible documentation confirms page metadata such as title and status, but does not establish dedicated OG fields. Parse the relevant HTML tags and validate them.
Q: Should I use og:url or canonical URL as the record ID?
Keep both values and define your canonicalization policy; flag disagreements for review.
Q: Can JavaScript alter metadata?
It can. Inspect the rendered DOM and the raw response when tag behavior matters for your audit.
Q: Does website metadata extraction require a separate proxy configuration with Nstdata Crawl?
A standard managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. Confirm any optional custom proxy selection and related usage charges in your account.
Q: Does HTTP 200 prove a website metadata extraction result is usable?
No. Check the nested task result, page status, expected fields, and body fidelity before using it in an AI or data workflow.
Kai Watanabe
Oct. 28th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
