import os
from nstdata_ai_crawl import Format, NstDataClient, ScrapeRequestDto
with NstDataClient(os.environ["NSTDATA_API_TOKEN"])as crawl: result = crawl.submit_scrape_task_sync(ScrapeRequestDto( url="https://docs.nstdata.io/docs/crawl", formats=[Format.MARKDOWN], timeout=60_000, onlyMainContent=True,)) markdown = result.data.get_markdown()ifnot markdown orlen(markdown.strip())<200:raise RuntimeError("Crawl returned empty or unexpectedly short Markdown")
defsplit_by_heading(markdown:str): path, lines, fenced =[],[],Falsefor line in markdown.splitlines():if line.strip().startswith("```"): fenced =not fenced
if line.startswith("#")andnot fenced and lines: body ="\n".join(lines).strip()if body:yield" / ".join(path), body
depth =len(line)-len(line.lstrip("#")) path = path[:depth -1]+[line[depth:].strip()] lines =[line]else: lines.append(line)if lines:yield" / ".join(path),"\n".join(lines).strip()sections =[(path, text)for path, text in split_by_heading(markdown)if text]ifnot sections:raise RuntimeError("没有生成可用的部分")
For a long section, split again on paragraph boundaries and keep the heading path. Do not claim every chunk is exactly a number of tokens unless you measured it with the embedding model’s tokenizer.
Step 4: Embed and index with Chroma
Chroma stores document IDs, text, embeddings, and metadata; see its collection API. This example computes embeddings explicitly so the query and documents use the same model and dimension.
import chromadb
from openai import OpenAI
openai_client = OpenAI()vector_db = chromadb.PersistentClient(path="./chroma-data")collection = vector_db.get_or_create_collection(name="url_demo")texts =[text for _, text in sections]embeddings = openai_client.embeddings.create( model="text-embedding-3-small",input=texts,).data
vectors =[item.embedding for item in embeddings]ids =[f"{page_hash[:16]}-{i}"for i inrange(len(texts))]collection.upsert( ids=ids, embeddings=vectors, documents=texts, metadatas=[{"source_url":"https://docs.nstdata.io/docs/crawl","heading_path": path,"page_hash": page_hash,}for path, _ in sections],)
For repeated URLs, replace old IDs associated with that URL after a new content version is accepted; otherwise an old page may remain searchable. The local Chroma path is convenient for a demo but needs backup, deployment, and concurrency decisions for production.
Step 5: Retrieve evidence for a question
Embed the question with the same model and retrieve a small top-k set:
The similarity distance is database-specific; do not treat it as a calibrated probability. Check the text and metadata that actually came back. For current tool-call behavior and response fields, see OpenAI’s function-calling guide.
Step 6: Generate an answer with citations
Treat crawled web text as untrusted input. It can contain instructions addressed to a model; those instructions are page content, not a change to your system policy. Pass it as quoted evidence and give the model no tools that can perform sensitive actions.
Do not show a canned answer as if it came from a live run. In an article example, label any sample response as illustrative and verify its facts against the cited page.
Step 7: Combine and test the demo
The snippets can be combined into one command that accepts a URL and question, but production code should split these responsibilities into functions and add structured logs. Test three source types: technical documentation, a normal article, and a page with dynamic content only if you have permission and a legitimate test case. Compare body completeness, retrieval evidence, citation URL, response time, and failures. Do not infer a general success rate from a handful of pages.
Useful tests include:
A question answered directly by one heading.
A question that requires a code block or table.
A question with no answer in the page, where the system should abstain.
A page that returns no content or a failed task.
A page update, where old chunks must no longer be active.
