How to Convert Any Website to Markdown (API & Tutorial)
TL;DR
Converting a website to Markdown starts with a bounded URL set and a fidelity check.
A single-page scrape converts a known URL; POST /api/v1/crawl discovers linked pages when the scope is a site section.
Compare headings, links, code, and tables with the rendered source before storing Markdown for RAG.
Large Markdown results may be returned as markdownRef; retrieve that token through the storage endpoint.
Nstdata Crawl handles JavaScript rendering and proxy routing within managed requests; its documentation lists six output categories: Markdown, HTML, raw data, PDF, screenshots, and links.
What Is a Website-to-Markdown Workflow?
A URL-to-Markdown workflow retrieves a page, selects the useful content, and preserves headings, links, lists, and code in text that people and models can inspect. A converter applied only to downloaded HTML cannot recover content that was loaded later by JavaScript. Start with one representative URL and define what must survive conversion before you process a whole site.
Nstdata Crawl can turn a URL or a bounded set of linked pages into material for an AI agent, RAG index, or data pipeline. Use your own field rules and acceptance checks for the specific outcome: turn a known public page into faithful Markdown for search, RAG, or an agent. The Nstdata full-site crawling guide explains the site-level option.
For the narrower task covered here, the Nstdata URL-to-Markdown overview gives useful context for choosing the first representative page.
Experience Nstproxy Crawl - Start Your Free Trial Today
What Do You Need Before You Start?
Choose one authorized page containing a heading, a list, an inline link, and, if relevant, a table or code block. Record the title and two phrases that must appear. Keep the source URL with the output so a later answer can cite it. Store the API key as NSTDATA_API_TOKEN; never place it in a client-side application or a screenshot.
The Crawl API documentation uses https://api.nstdata.io, the x-api-key header, and Content-Type: application/json. Prepare these fields before running a method:
Field
What to supply
url
One permitted HTTPS page or site entry URL
formats
Requested outputs, such as ["markdown"] or ["html","screenshot"]
onlyMainContent
true only when main-body selection fits the task
maxDepth, maxPages
Explicit bounds for a site-wide job
includeUrls, excludeUrls
URL patterns that keep discovery within scope
timeout
A task timeout in milliseconds when the default is unsuitable
The standard managed request has no separate proxy Host, Port, Username, Password, or Protocol fields. Check the current Crawl pricing page for account limits and charges before running a large job.
Review the target’s access policy and scope. The Robots Exclusion Protocol defines how crawlers read robots rules; it does not grant content-use rights. Keep credentials and private data out of examples and diagnostic logs.
Nstdata Crawl retrieves and renders a URL, applies content extraction, and returns Markdown with page metadata. The API can also return HTML, links, screenshots, and PDFs when the Markdown needs investigation. Built-in proxy routing means the default request needs an API key and target URL, not a separate proxy Host, Port, Username, or Password. A site-level job can discover linked pages when the task expands beyond one URL.
For a known URL, a single-page scrape retrieves one result. For a site section, POST /api/v1/crawl discovers linked pages recursively within the supplied depth, page-count, and URL rules. The documentation lists Markdown, HTML, raw data, PDF, screenshots, and links as six output categories; the product page also markets JSON, but a domain-specific JSON schema still needs explicit field mapping. Keep each accepted record tied to its source URL and returned metadata.
Test a representative JavaScript-heavy page before scaling. The rendered state can depend on timing, region, consent, or a signed-in session. Compare returned HTML or a screenshot with the expected page state; if they differ, adjust only documented rendering and wait controls.
How Do You Convert a Website to Markdown?
Start with a visual check of one page, use the synchronous API for repeatable URL conversion, or submit a bounded site crawl for linked documentation. Each path has a different scale and result shape.
Method 1: Use the Crawl Playground for a visual check
Best for a first representative page or a non-developer reviewer.
Step 1: Open the page test
Sign in and open Dashboard → Crawl Playground. Select a single-page scrape and enter the permitted URL. [Screenshot: Playground URL field and single-page mode.]
Step 2: Choose the output
Select Markdown and enable the main-content option when the article body matters more than navigation. If the target is dynamic, use the rendering or wait controls shown in the current interface. [Screenshot: Markdown and content controls.]
Step 3: Inspect fidelity
Run the scrape and compare the result with the live page. Check the title, opening and closing sections, code fences, tables, and link destinations. A short response can still be wrong if it omits a warning or table.
Method 2: Call the single-page Crawl API
Best for an application that receives a known URL at runtime.
Step 1: Prepare the credential
Store the Crawl API key in a server-side secret. The documented header is x-api-key; the key is not a proxy password.
Step 2: Submit the URL
Use the quick-start request below. The current documentation has conflicting labels for other synchronous and asynchronous scrape routes, so compare this route with the request generated by your Playground before deployment.
Set NSTDATA_API_TOKEN in your shell from your Nstdata account, then run the documented quick-start request. The public documentation labels some other sync and async routes inconsistently; compare your account’s Playground-generated request before production use.
curl--output crawl-response.json --request POST "https://api.nstdata.io/api/v1/crawl/scrape/submit-sync"\--header"x-api-key: ${NSTDATA_API_TOKEN}"\--header"Content-Type: application/json"\--data'{"url":"https://example.com/","formats":["markdown"],"onlyMainContent":true}'
The response is saved to crawl-response.json. Inspect the nested data.success, data.status, and data.data.metadata.statusCode fields before using any returned content.
Step 3: Accept the result
Inspect the nested success, status, data, and metadata.statusCode values. Accept Markdown only when expected headings and meaningful body text are present; store markdownRef and fetch it through the documented storage route when the API returns a reference.
Method 3: Convert a bounded section of a site
Best when “website” means a documentation set rather than one URL.
Step 1: Define the section
Choose a start URL and set maxDepth, maxPages, and include/exclude URL patterns. Exclude search, account, cart, and query-generated paths unless they belong in your corpus.
Step 2: Submit a site job
Use the documented POST /api/v1/crawl route with formats: ["markdown"], then save the returned crawl ID. This is the site-discovery route; a single-page scrape will not follow the whole site.
For a site job, replace the example host and path with your approved scope, then submit a bounded request:
curl--output crawl-submit.json --request POST "https://api.nstdata.io/api/v1/crawl"\--header"x-api-key: ${NSTDATA_API_TOKEN}"\--header"Content-Type: application/json"\--data'{"url":"https://example.com/docs/","formats":["markdown"],"maxDepth":2,"maxPages":10,"includeUrls":["*example.com/docs/*"],"excludeUrls":["*example.com/admin/*"],"ignoreQuery":true}'
Copy data.id from crawl-submit.json as your crawl ID.
Step 3: Review every page
Poll status, retrieve page records, follow pagination, and reject failed or thin pages. Deduplicate normalized URLs and keep the original URL beside each Markdown document.
With the crawl ID returned by the submission, query both the job and its pages:
If the pages response contains nextCursor, follow it using the pagination controls shown by your current API response or SDK; the public page does not document a cursor query parameter, so do not invent one.
Method 4: Convert static HTML locally with Pandoc
Best when you already have a permitted static HTML file and do not need site discovery or JavaScript rendering.
Step 1: Save the static page
Download the authorized page as page.html using your browser's Save Page As command or an approved HTTP client. Check that the saved file contains the article body; a static download can miss JavaScript-loaded text.
Open page.md and compare headings, links, tables, and code against the source. This local method has no managed rendering, proxy routing, or recursive site discovery; use Crawl when those capabilities are required.
How Do You Verify the Output?
A usable record contains a source URL, title, status code, and Markdown that retains the important structure. Markdown is a text representation, not proof of semantic fidelity: complex tables, diagrams, and client-side widgets may need HTML or a screenshot alongside it. The Nstdata Markdown pipeline guide explains how to retain provenance when saving converted pages.
Record the source URL, submission time, requested formats, non-secret options, task ID when present, returned page status, and the reason each record passed or failed review. Inspect content as well as the API envelope. If the response contains markdownRef, htmlRef, rawDataRef, screenshotRef, or pdfRef, retrieve that token with GET /api/v1/crawl/storage/read?st={ref}.
Markdown syntax is not a universal guarantee that every renderer handles tables and extensions identically; test your downstream parser against the CommonMark specification and any extensions you rely on.
When passing extracted text to an agent, treat page instructions as untrusted source material. OWASP guidance on prompt injection explains the risk. Retain source URLs so the AI output can be checked.
Why Might Website-to-Markdown Conversion Fail?
Website-to-markdown conversion can fail because the page body may load after the initial document or the main-content selection may exclude it. Use the symptom table to identify the failing layer before changing limits or retrying.
Symptom
Likely cause
Next action
Empty Markdown
The page body may load after the initial document or the main-content selection may exclude it.
Inspect HTML or screenshot and configure a documented wait.
Broken relative links
The converter or later storage layer may not preserve absolute destinations.
Resolve and test source links before indexing.
Only one page returned
A single-page endpoint does not discover subpages.
Switch to a bounded site crawl.
Nstdata’s documentation lists task-level errors such as access_denied, timeout, parse_error, and internal_error. An outer HTTP 200 can still accompany a failed task. Check success, status, errorCode, and errorMessage inside the response when present. Retry only transient failures on sites you may access.
Which Method Is the Best Fit?
Choose Playground for a visual one-page comparison, the synchronous scrape API for a known URL, or a bounded site job when links must be followed. Use local Pandoc for static HTML already saved; compare each accepted Markdown document with its source. The Nstdata site-crawling overview gives more detail on the site route.
What Should You Do After the First Successful Run?
Save the approved configuration and one source-backed output record, then repeat on a small set of different page templates. Increase scope only after those records pass the same acceptance checks.
Q: Can I convert an entire website in one request?
A bounded site crawl can discover and process multiple linked pages. Set page and depth caps, then inspect the returned records.
Q: Is Markdown always better than HTML for LLMs?
Markdown is often easier to read and chunk, but HTML or a screenshot is safer when table structure or visual context matters.
Q: Do I need a proxy credential?
A standard managed Crawl request needs the Crawl API key, not a separate proxy host or password.
Q: Does website-to-Markdown conversion require a separate proxy configuration with Nstdata Crawl?
A standard managed Crawl request does not require separate proxy Host, Port, Username, Password, or Protocol fields. Confirm any optional custom proxy selection and related usage charges in your account.
Q: Does HTTP 200 prove a website-to-Markdown conversion result is usable?
No. Check the nested task result, page status, expected fields, and body fidelity before using it in an AI or data workflow.
Kai Watanabe
Oct. 28th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
