Introduction: Why build a knowledge base from a docs site?
Nstdata Crawl is the collection layer in this workflow; its product overview describes page and site retrieval for downstream data pipelines.
A SaaS documentation site often answers the same questions from customers, support, and developers: how to authenticate, which SDK version to install, what a parameter does, or how to troubleshoot an error. A retrieval-powered assistant can make those pages easier to query in natural language, provided the index preserves which product and version each answer came from.
The goal is an evidence-backed search layer, not an automatic truth engine. The system can retrieve an outdated page or misread a parameter table if ingestion and evaluation are weak. Nstdata Crawl can supply page content as Markdown and support bounded site discovery; your application remains responsible for selection, permissions, indexing, update policy, and answer behavior. The official Crawl documentation describes the acquisition layer and output formats.
What you will build
Public docs pages
↓ crawl and validate
Markdown + provenance metadata
↓ normalize and chunk
Embeddings + vector records
↓ retrieve relevant passages
Assistant answer with source URLs and version labels
A user could ask, “Which header authenticates the current API?” The assistant should retrieve a page from the current API version, quote the relevant evidence, and show its URL. If the corpus contains both an old and current version and the question does not identify which one, the assistant should ask or state which version it used.
Step 1: Audit the documentation website
Start with the site’s information architecture, not the crawler. Review its sitemap, navigation, version selector, locale paths, canonical links, and authentication boundaries. Make a URL table with a decision and owner:
Page group
Priority
Example metadata
Ingestion rule
Getting started
High
product, language, version
include current version
API reference
High
endpoint, version, SDK
preserve parameter tables
Troubleshooting
High
component, error code
keep code and warning context
Pricing
Medium/high
currency, effective date
refresh more often
Changelog archive
Depends
release date, status
separate from current docs
Login-only internal docs
Separate project
ACL and owner
do not expose through public index
Identify high-value documentation pages
A small pilot should usually cover Getting Started, API Reference, Authentication, SDK guides, Troubleshooting, and FAQ. These categories answer concrete support questions and reveal parsing issues early. Keep marketing pages out unless the assistant is meant to answer product-positioning questions too.
Handle multiple versions and languages
Do not merge /v1/ and /v2/ content into a single undifferentiated set. Store product, version, language, status (current/deprecated), and canonical_url. Apply a retrieval filter from the user’s selected product and version. Keep localized pages distinct, because translation differences may affect parameters and examples.
Step 2: Crawl the public docs into Markdown
Use site-level Crawl from the docs root with page and depth limits. Include only docs paths, exclude login/search routes and unrelated assets, and inspect the discovered URL list before indexing. Do a small test first with a representative static page, a code-heavy API page, and a page that uses client-side rendering.
The documented Python SDK supports a CrawlRequestDto with fields such as url, formats, maxDepth, maxPages, includeUrls, excludeUrls, ignoreQuery, and onlyMainContent:
import os
from nstdata_ai_crawl import CrawlRequestDto, Format, NstDataClient
with NstDataClient(os.environ["NSTDATA_API_TOKEN"])as client: task = client.submit_crawl_task(CrawlRequestDto( url="https://docs.example.com/", formats=[Format.MARKDOWN], maxDepth=3, maxPages=250, includeUrls=["*docs.example.com/*"], excludeUrls=["*docs.example.com/search*","*docs.example.com/login*"], ignoreQuery=True, onlyMainContent=True, timeout=60_000,)) crawl_id = task.idprint(crawl_id)
This is the submission step; do not mistake a returned task ID for finished pages. Poll task status, page through the results, and fetch any referenced Markdown artifact through the documented storage endpoint. Keep failed URLs in the run report. For a practical crawl-to-vector-store implementation, see the site’s Pinecone workflow guide.
Step 3: Clean and organize the content
Remove repeated navigation or cookie banners only when they are clearly boilerplate. Preserve headings, code language labels, parameter tables, admonitions, links, and version notices. A before/after example should show why the cleaning rule exists:
Before: [Docs Home] [API] [SDK] | Authentication | Use the x-api-key header... | Related: SDKs
After: ## Authentication
Send the API token in the `x-api-key` request header...
The “Before” row is illustrative, not a crawl result. For each real page, store the extracted body and its metadata in a structured record:
Populate crawled_at at runtime; the date above is a schema example. Use a stable hash of normalized Markdown to distinguish a real content change from a repeated crawl.
Step 4: Chunk and embed technical documentation
Chunk by headings first. Keep an API method, endpoint, parameter table, response schema, and relevant code sample together when they describe one operation. If a section exceeds the embedding input limit, split it at a paragraph or code-example boundary and copy the heading path into each chunk’s metadata.
Use one embedding model for a collection generation and record its name. Store source_url, title, version, language, section_path, and crawled_at with each record. The vector itself is not a replacement for the text or its citation metadata. See the OpenAI embeddings guide for the embedding request contract.
Step 5: Store and retrieve with a version filter
This example uses Chroma for a local pilot; its Python client and collection reference describes local and persistent collection workflows. Production deployment needs an explicit backup and permission plan. For a managed alternative, review Pinecone’s record upsert documentation. Upsert chunks with stable IDs, then filter queries by product and selected version.
import chromadb
client = chromadb.PersistentClient(path="./docs-index")collection = client.get_or_create_collection("company_docs")collection.upsert( ids=["auth-v2-001"], documents=["Send the API token in the x-api-key header."], embeddings=[embedding],# generated by your chosen embedding model metadatas=[{"source_url":"https://docs.example.com/v2/authentication","title":"Authentication","product":"Example API","version":"v2","language":"en","section_path":"API Reference / Authentication",}],)
At query time, retrieve only the permitted product/version and pass the source link into the answer prompt. Test direct API questions, version-specific questions, and unanswerable questions. When there is not enough retrieved evidence, the assistant should decline to guess.
Step 6: Build a documentation Q&A assistant
The application flow is: classify the request, apply user-allowed filters, retrieve passages, construct a context with URLs and versions, ask the model for a grounded answer, then validate the returned citations. The response should show a concise answer, links to the source sections, and a version label. Log the retrieved IDs and prompt version so an incorrect answer can be traced.
Use API-authentication questions, SDK-installation questions, and error-code questions in the test set. For every test, a reviewer should confirm that the cited page contains the claim. A matching URL is not enough if it points to an old version.
Handle missing, conflicting, and outdated docs
If no page covers a question, say that the indexed docs do not confirm an answer. If versions disagree, retrieve the version selected by the user or ask a clarifying question. If a page has been superseded, mark it deprecated and filter it out of the default current-docs collection while retaining it for explicit historical queries.
Step 7: Keep the docs index updated
Run discovery on a schedule or trigger it from the documentation build system, then compare normalized content hashes. Re-embed changed pages and remove stale chunk IDs only after the new version validates. Treat transient errors as failures, not deletions. Keep a last-known-good generation available while a new crawl is being processed.
Security and access control
Public and internal documentation need different access controls. Do not assume that a crawler’s API key enforces end-user document permissions. For private pages, confirm the source owner authorizes collection, protect credentials, separate indexes or enforce document-level ACL filters, and log access decisions. Never index secrets or customer data into a shared collection by default. Consult your organization’s security team before ingesting confidential material.
NIST’s access-control material is a useful starting point for designing policy boundaries; see its Access Control Systems guidance. The applicable controls depend on your deployment and data classification.
Costs, maintenance, and common problems
Budget for crawl requests, proxy usage if applicable, embedding tokens, vector storage, model queries, and engineering maintenance. Re-crawl frequency and document changes often matter more than the first import. Do not quote a per-page total without measuring how many chunks each page produces and how often it changes; check current Crawl pricing before forecasting.
Typical issues include mixed API versions, repeated nav chunks, broken code fences, outdated records remaining after an update, citations that point to a generic docs home page, and content that users cannot access. Fix the data contract first: stable IDs, canonical URLs, version metadata, an allowlist, and a freshness job.
Launch checklist
Define public versus private source boundaries.
Keep a current URL inventory and explicit version policy.
Preserve code, tables, warnings, URLs, and version metadata.
Validate answers against real support and developer questions.
Enforce user permissions before returning retrieved context.
Monitor change detection, stale chunks, failed URLs, and citation correctness.
A company knowledge base from a docs site succeeds when it keeps the source context intact. Begin with a small public corpus, model product and version metadata explicitly, and require the assistant to cite the page it used. Add private or multi-tenant content only after retrieval filters and access controls have been tested independently.
Q: How do I build a knowledge base from a website?
Select and crawl permitted pages, preserve source metadata, clean and chunk the content, embed it, store it in a searchable index, and validate answers with citations.
Q: Can an AI knowledge base use a documentation website?
Yes, if the documentation can be lawfully accessed and its version, language, and source URL remain attached to indexed passages.
Q: How do I create an AI assistant for company docs?
Connect a retriever to a curated documentation index, apply user permissions and version filters, and instruct the model to answer from retrieved passages with citations.
Q: How should I handle API versions in RAG?
Store the API version as metadata, keep versions distinct, and filter retrieval to the version relevant to the user’s question.
Q: Can I crawl internal documentation behind login?
Only with explicit source authorization and a design that enforces the same access boundaries for every user of the resulting index.
Understand seven dimensions of RAG data quality and how to validate crawled pages, compare clean and raw inputs, preserve provenance, and diagnose retrieval failures. It separates extraction, retrieval, and generation errors so teams can debug the right stage.
Lena Zhou
Oct. 20th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
