Introduction: Documentation is structured data, even when it is published as HTML
Nstdata Crawl is the collection layer in this workflow; its product overview describes page and site retrieval for downstream data pipelines.
A documentation site is built for readers navigating pages, versions, and code examples. An LLM needs the same relationships to survive extraction. If a crawler returns an API parameter without its endpoint, code without its language label, or an old v1 page next to a current v2 page with no version metadata, the resulting context can be misleading even if every URL returned successfully.
This guide covers documentation discovery, crawling, Markdown validation, and metadata. It does not attempt to teach a full vector search system. For the later embedding stage, see the separate website-to-Pinecone walkthrough. Nstdata Crawl supports Markdown, HTML, raw data, links, PDF, and screenshot output; the current Crawl docs describe the trade-offs and task interfaces.
What makes documentation LLM-ready?
A useful documentation record includes complete content, meaningful heading hierarchy, intact code and tables, stable source URL, version, language, crawl timestamp, and a content hash. “Markdown conversion succeeded” is not the same as “the LLM can answer the intended question.” Validate the specific information your downstream use case needs.
For API reference pages, keep the method, endpoint, authentication details, parameter names, types, defaults, response fields, and error behavior in the same local context. For guides, preserve the sequence of steps and warnings. For changelogs, preserve release dates and status labels.
How documentation crawling differs from general web scraping
Docs often use frameworks such as Docusaurus, Nextra, VitePress, or MkDocs, but their rendering modes differ by deployment. Do not assume a JavaScript browser is always required. Inspect the output from the page you need: if key text is absent in the retrieved result, then test browser rendering or a documented wait. The public docs for Docusaurus, VitePress, and MkDocs describe their publishing systems; the deployed site’s behavior still needs to be checked directly.
Step 1: Discover documentation URLs
Use the sitemap as one source and site links as another. A sitemap may include pages the docs navigation no longer exposes, while link discovery may miss orphaned pages that are only listed in a sitemap. Compare the resulting URL sets and classify each path before crawling.
Sitemap discovery versus link crawling
Method
Useful for
Blind spot to check
Sitemap
declared page inventory and lastmod hints
stale or omitted entries
Link discovery
pages reachable from the docs navigation
orphaned pages or deep links
Curated URL list
a controlled pilot or release set
manual maintenance burden
The Sitemaps protocol defines the XML format, but a lastmod field is publisher-provided metadata, not an independent guarantee of content change. Use it to prioritize checks, then compare page content.
Exclude irrelevant and duplicate URLs
Normalize hostnames, strip fragments, and decide which query parameters affect page identity. Exclude search results, login pages, tracking variants, print routes, duplicate locale paths, assets, and unrelated marketing pages. Keep both versioned docs and localized pages only when they are within scope. Record why a page was excluded so a coverage gap is explainable.
Step 2: Crawl docs pages into Markdown
Run a small pilot first. For a known page, single-page scrape avoids site-discovery variables. For a docs section, a bounded site crawl can discover links with a depth and page limit. The current Python SDK example uses NstDataClient, CrawlRequestDto, and Format.MARKDOWN:
import os
from nstdata_ai_crawl import CrawlRequestDto, Format, NstDataClient
with NstDataClient(os.environ["NSTDATA_API_TOKEN"])as client: task = client.submit_crawl_task(CrawlRequestDto( url="https://docs.example.com/", formats=[Format.MARKDOWN], maxDepth=2, maxPages=80, includeUrls=["*docs.example.com/*"], excludeUrls=["*docs.example.com/search*","*docs.example.com/login*"], ignoreQuery=True, onlyMainContent=True, timeout=60_000,))print("task id:", task.id)
This code submits the job; it does not show the completed page collection. Poll status, retrieve the pages list, follow its cursor, and read referenced Markdown artifacts as documented. The current Crawl guide lists GET /api/v1/crawl/status/{crawlId} and /pages/{crawlId}; webhooks are not currently supported, so make polling part of your worker’s retry and timeout design.
JavaScript-rendered documentation
Check whether a known phrase, code block, or page title is present in returned Markdown. Compare against the rendered page in a browser. If the content is absent, test JavaScript rendering and a selector wait when supported. Avoid enabling rendering globally without a reason: it can add time and makes it harder to tell whether a failure came from discovery, rendering, or extraction.
Step 3: Preserve headings, code blocks, and tables
Markdown should preserve H1/H2/H3, code fences with language labels, tables, lists, warnings, and internal links. A useful validation sample might contain:
## Create a tokenSend the token in the `x-api-key` header.
| Field | Type | Required ||---|---|---||`project_id`| string | yes |```python
client = Client(token=token)
This is a synthetic fixture for checking formatting, not captured output from a live page. After a crawl, check that the opening and closing code fences match and that table headers and cell counts remain consistent.
Preserve endpoint and parameter context
An API method is hard to use if its parameter table gets separated from the endpoint. Store the nearest parent headings with every chunk. Preserve warnings such as “deprecated” or “required for this version”; they often change whether an example is safe to copy.
Handle code samples in multiple languages
Keep the code language label, SDK/version, endpoint or feature name, and surrounding explanatory text. Do not concatenate Python and Node.js examples into a single undifferentiated chunk. If a code block is extremely long, split at logical example or function boundaries and repeat the relevant heading path.
Step 4: Handle documentation versions and localization
Treat /v1/, /v2/, “latest,” and archived routes as explicit states. Store version, status, language, and canonical_url separately from body text. latest is a moving pointer; capture the resolved version if the site identifies it. Keep translations separate and avoid merging examples across locales without review.
Step 5: Normalize metadata
For each page, retain:
{"source_url":"https://docs.example.com/v2/reference/users","canonical_url":"https://docs.example.com/v2/reference/users","title":"Users API","section_path":"API Reference / Users","version":"v2","language":"en","crawled_at":"runtime timestamp","content_hash":"hash of normalized body"}
Normalize URL identity consistently and hash the normalized body rather than the full HTML, so a layout change does not force unnecessary re-embedding. Never fabricate source update dates when a page does not publish them.
Step 6: Validate crawl quality
Create automated checks for empty body, unexpectedly short text, repeated boilerplate, missing title, unclosed code fences, malformed tables, broken canonical URL, duplicate normalized content, and absent version metadata. Then manually sample pages by type: overview, API reference, code-heavy tutorial, and a dynamic page if the docs use one.
A content completeness check should be tied to the task: for an API page, compare expected endpoint names and important fields; for a tutorial, check that headings and code blocks are present. A green HTTP status does not prove the extracted content is complete.
Step 7: Store documentation for LLM use
Save crawl artifacts as Markdown files or JSONL during the pilot so a reviewer can inspect them. A database becomes useful when you need incremental updates, page-level status, permissions, or query access. A JSONL record should keep page text and metadata together, with chunk records referencing the page ID. See the Crawl-to-Pinecone implementation for the next ingestion stage.
Large sites, monitoring, and responsible access
Bound the first crawl with depth, page caps, and URL filters. Monitor task status and per-page failures; continue from a checkpoint rather than restarting every page after one error. Use reasonable rates and respect site terms, robots guidance, and access permissions. Do not try to bypass logins or controls for content you are not authorized to collect.
Metadata completeness: records with required source/version fields / accepted records.
These metrics are more useful than counting successful requests alone. Keep a sample manifest and rerun it after crawler, docs-site, or extraction changes.
Common mistakes and final checklist
Common mistakes include crawling only the homepage, excluding the old/current version relationship, indexing search pages, losing code labels, mixing languages, accepting empty client-rendered pages, and treating failed requests as deleted content.
Before sending documentation to an LLM, verify URL discovery, access scope, page-level success, content structure, version and language metadata, duplicate handling, and update behavior. Keep the original source URL attached through chunking so every answer can cite it.
Crawling documentation for LLM context is a structured-data quality task. Discover pages from multiple signals, bound the crawl, preserve the technical relationships inside each page, and verify the result against representative API questions. Once the Markdown passes those checks, it can feed chunking, search, RAG, or an agent tool.
Start with a sitemap and navigation inventory, filter to permitted docs paths, submit a bounded crawl, retrieve each result, and validate the Markdown before storing it.
Q: What is the best format for LLM documentation?
Markdown is a practical text format when it preserves headings, code, tables, links, and source metadata; HTML can be better when custom parsing is required.
Q: How do I convert docs to Markdown?
Use a page extraction tool that returns Markdown, then check the result for missing sections, malformed tables, and code-block integrity.
Q: Do Docusaurus docs require JavaScript rendering?
Not necessarily; inspect the target page output first and enable rendering only when required content is absent from the non-rendered result.
Q: How do I preserve code blocks for RAG?
Keep fenced blocks with their language labels and nearby heading context, and split long examples only at logical boundaries.
Best practices for discovering and crawling documentation URLs, preserving code and tables, managing versions, and validating LLM-ready Markdown. It includes a practical validation checklist for Markdown output and page coverage.
Ivy Lin
Oct. 20th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
