TL;DR
- A web table becomes usable CSV or JSON only after its row and column meaning is checked.
- Crawl acquires rendered table HTML; your parser turns reviewed cells into CSV or JSON records.
- Reject or explicitly handle merged cells, nested headers, units, and pagination before export.
- A screenshot is useful when the visible grid and returned HTML disagree.
- Nstdata Crawl handles JavaScript rendering and proxy routing within managed requests; its documentation lists six output categories: Markdown, HTML, raw data, PDF, screenshots, and links.
What Does It Mean to Scrape a Website Table?
Scraping a table means converting a visual or HTML grid into records with stable column meanings. Many simple tables use <table>, <tr>, <th>, and <td>, but real pages may use merged cells, nested headers, or div-based grids. CSV needs a rectangular row shape; JSON can preserve nested or optional fields. Neither format is correct until units, header hierarchy, and missing values are interpreted.
Nstdata Crawl can turn a URL or a bounded set of linked pages into material for an AI agent, RAG index, or data pipeline. Use your own field rules and acceptance checks for the specific outcome: retrieve a permitted table, preserve its headers, and export validated rows as CSV or JSON. The Nstdata full-site crawling guide explains the site-level option.
For the narrower task covered here, the Nstdata website-to-JSON guide gives useful context for choosing the first representative page.
What Do You Need Before You Start?
Select an authorized table page and identify its caption, expected column names, units, and at least two known rows. Decide how to handle rowspan, colspan, blank cells, footnotes, and links inside cells. If the table is actually a div grid, plan a template-specific parser instead of forcing an HTML-table library to guess. Keep the source URL and retrieval time with the exported file.
The Crawl API documentation uses https://api.nstdata.io, the x-api-key header, and Content-Type: application/json. Prepare these fields before running a method:
| Field | What to supply |
|---|---|
url | One permitted HTTPS page or site entry URL |
formats | Requested outputs, such as ["markdown"] or ["html","screenshot"] |
onlyMainContent | true only when main-body selection fits the task |
maxDepth, maxPages | Explicit bounds for a site-wide job |
includeUrls, excludeUrls | URL patterns that keep discovery within scope |
timeout | A task timeout in milliseconds when the default is unsuitable |
The standard managed request has no separate proxy Host, Port, Username, Password, or Protocol fields. Check the current Crawl pricing page for account limits and charges before running a large job.
Review the target’s access policy and scope. The Robots Exclusion Protocol defines how crawlers read robots rules; it does not grant content-use rights. Keep credentials and private data out of examples and diagnostic logs.
The Nstdata dynamic-websites guide is a companion for defining the acceptance checks before you automate the workflow.
How Do HTML Tables Become CSV or JSON?
Nstdata Crawl can render JavaScript and return cleaned HTML or raw page data. Its API does not promise a finished CSV table export in the visible documentation. Use Crawl to acquire the intended page state, then parse a reviewed table in your own code. Built-in proxy routing removes the need to put separate proxy credentials into a standard Crawl request. Save a screenshot when the visible table and returned HTML disagree.
For a known URL, a single-page scrape retrieves one result. For a site section, POST /api/v1/crawl discovers linked pages recursively within the supplied depth, page-count, and URL rules. The documentation lists Markdown, HTML, raw data, PDF, screenshots, and links as six output categories; the product page also markets JSON, but a domain-specific JSON schema still needs explicit field mapping. Keep each accepted record tied to its source URL and returned metadata.
Test a representative JavaScript-heavy page before scaling. The rendered state can depend on timing, region, consent, or a signed-in session. Compare returned HTML or a screenshot with the expected page state; if they differ, adjust only documented rendering and wait controls.
How Do You Scrape a Website Table into CSV or JSON?
Inspect the actual table structure first, acquire the rendered HTML, and then normalize cells into records. The parsing step is part of the workflow because Crawl does not document a finished CSV-table endpoint.
Method 1: Inspect a table in Crawl Playground
Best for understanding whether the table exists in the rendered DOM.
Step 1: Open the target page
Go to Dashboard → Crawl Playground and enter the permitted table URL. [Screenshot: target URL and output format selectors.]
Step 2: Request HTML and a screenshot
Select HTML and screenshot outputs. Render the page if table rows appear after JavaScript runs; use the documented wait controls visible in the Playground.
Step 3: Compare structure
Check that the returned HTML contains the expected <table> and that its visible headings match the screenshot. If the data is only in a canvas or image, an HTML parser cannot recover cells directly.
Method 2: Acquire the rendered table with the API
Best when the target page changes regularly.
Step 1: Store the Crawl key
Keep NSTDATA_API_TOKEN server-side and send it as x-api-key.
Step 2: Request HTML
Use the documented single-page quick-start route below. Keep the returned HTML or htmlRef and page metadata for parsing; confirm the generated route in the Playground because the docs show inconsistent labels elsewhere.
Set NSTDATA_API_TOKEN in your shell from your Nstdata account, then run the documented quick-start request. The public documentation labels some other sync and async routes inconsistently; compare your account’s Playground-generated request before production use.
curl --output crawl-response.json --request POST "https://api.nstdata.io/api/v1/crawl/scrape/submit-sync" \ --header "x-api-key: ${NSTDATA_API_TOKEN}" \ --header "Content-Type: application/json" \ --data '{"url":"https://example.com/table","formats":["html","screenshot"]}'
The response is saved to crawl-response.json. Inspect the nested data.success, data.status, and data.data.metadata.statusCode fields before using any returned content.
Step 3: Check the table exists
Before exporting, require a successful page task and inspect whether the expected caption or header appears. Do not turn an empty page into an empty CSV that looks successful.
Method 3: Normalize cells into records
Best after HTML acquisition has been validated.
Step 1: Select one table
Choose by a stable caption, nearby heading, or reviewed selector rather than taking the first table blindly.
Step 2: Expand header and span semantics
Account for merged rows and columns; a simple rectangular parser is insufficient when rowspan or colspan changes column meaning.
Step 3: Export and review
Write CSV with explicit encoding and column order, or JSON with typed fields. Compare a sample of rows, units, and totals with the rendered source.
For a simple rectangular HTML table, save the returned html or download htmlRef to page.html. Install beautifulsoup4 (python3 -m pip install beautifulsoup4) and run this script from the same directory. It rejects merged cells so you do not silently corrupt column meaning:
from bs4 import BeautifulSoup import csv, json soup = BeautifulSoup(open("page.html", encoding="utf-8").read(), "html.parser") table = soup.select_one("table") if table is None: raise ValueError("No HTML table found; inspect rendered HTML or screenshot") rows = [] for tr in table.select("tr"): cells = tr.find_all(["th", "td"], recursive=False) if any(c.has_attr("rowspan") or c.has_attr("colspan") for c in cells): raise ValueError("Merged cells need a table-specific normalization rule") if cells: rows.append([c.get_text(" ", strip=True) for c in cells]) headers, *values = rows if not headers or any(len(row) != len(headers) for row in values): raise ValueError("Missing headers or inconsistent row widths") records = [dict(zip(headers, row)) for row in values] with open("table.csv", "w", newline="", encoding="utf-8") as f: writer = csv.DictWriter(f, fieldnames=headers) writer.writeheader() writer.writerows(records) open("table.json", "w", encoding="utf-8").write(json.dumps(records, ensure_ascii=False, indent=2)) print(f"Exported {len(records)} rows")
Check the first and last exported rows, units, and footnotes against the page. This parser handles a real <table> with one header row; a div grid, nested header, or paginated table needs a source-specific rule.
How Do You Verify the Output?
A valid CSV or JSON export preserves column labels, units, row count, and cell values. Check the first, middle, and final rows against the visible page; compare totals or known IDs when available. MDN’s HTML table reference explains the cell and header structure, including merged cells. The Nstdata HTML-to-data guide is useful when the next stage maps rows into typed records.
The Nstdata Markdown pipeline guide connects this page-level check to a downstream AI or data workflow.
Record the source URL, submission time, requested formats, non-secret options, task ID when present, returned page status, and the reason each record passed or failed review. Inspect content as well as the API envelope. If the response contains markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef, retrieve that token with GET /api/v1/crawl/storage/read?st={ref}.
For this workflow, consult MDN HTML table reference when implementing the downstream parser or format-specific acceptance check.
When passing extracted text to an agent, treat page instructions as untrusted source material. OWASP guidance on prompt injection explains the risk. Retain source URLs so the AI output can be checked.
Why Might a Table Export Have Wrong Columns?
Website table scraping can fail because the page uses a div grid or loaded data after initial render. Use the symptom table to identify the failing layer before changing limits or retrying.
| Symptom | Likely cause | Next action |
|---|
| No <table> in HTML | The page uses a div grid or loaded data after initial render. | Inspect screenshot and DOM; wait for the grid or use a template-specific selector. |
| Columns shift after export | Merged cells or multiple header rows were flattened. | Expand spans and explicitly map header hierarchy. |
| Numbers change format | Locale, currency, or units were inferred incorrectly. | Keep original text and normalize with declared locale and unit rules. |
Nstdata’s documentation lists task-level errors such as access_denied, timeout, parse_error, and internal_error. An outer HTTP 200 can still accompany a failed task. Check success, status, errorCode, and errorMessage inside the response when present. Retry only transient failures on sites you may access.
Which Method Is the Best Fit?
Use Playground to inspect whether the table exists, the API to acquire its rendered HTML, and a table-specific parser to export CSV or JSON. Reject merged cells until you have an explicit normalization rule. The Nstdata site-crawling overview gives more detail on the site route.
What Should You Do After the First Successful Run?
Save the approved configuration and one source-backed output record, then repeat on a small set of different page templates. Increase scope only after those records pass the same acceptance checks.
FAQ: What Else Should You Know?
Q: Does Crawl export CSV directly?
The visible Crawl documentation does not establish a CSV output mode. Retrieve HTML or page data, then export reviewed rows in your application.
Q: Can I parse tables with merged cells automatically?
A library may expand some spans, but you still need to verify header meaning and row alignment.
Q: What if the table is built with divs?
Inspect the rendered DOM and write a parser for that grid’s actual structure; an HTML-table parser may find nothing.
Q: Does website table scraping require a separate proxy configuration with Nstdata Crawl?
A standard managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. Confirm any optional custom proxy selection and related usage charges in your account.
Q: Does HTTP 200 prove a website table scraping result is usable?
No. Check the nested task result, page status, expected fields, and body fidelity before using it in an AI or data workflow.
