Website to Vector Database: Crawl, Chunk, Embed in One Flow
TL;DR
A website-to-vector-database pipeline stores searchable chunks and provenance metadata; it does not put a whole website into one vector.
Crawl a bounded set of pages, validate the Markdown, split it into stable chunks, embed each chunk, then upsert it to a Pinecone namespace.
The embedding model’s output dimension must match the Pinecone dense index, and repeated ingestion should reuse deterministic IDs instead of creating duplicates.
This is a build-oriented companion to Nstdata’s existing Crawl-to-Pinecone overview; the focus here is the data contract and safe refresh path.
Nstdata Crawl is the collection layer in this workflow; its product overview describes page and site retrieval for downstream data pipelines.
This tutorial turns a public documentation site into searchable records in Pinecone. The flow is URL → Crawl → Markdown → chunks → embeddings → Pinecone → query. It stops at retrieval; a chat UI, user permissions, and production monitoring are separate application layers.
A vector database does not normally store a website as one giant document. Each record is a vector for a passage, plus an ID and metadata that let you find the source and update it later. Nstdata Crawl supplies the page collection and Markdown conversion stage; Pinecone stores and retrieves vectors. For current Crawl endpoints and task responses, use the Nstdata Crawl API reference.
How website content becomes vector data
Step
Example value
Why it is stored
Page
https://docs.example.com/auth
Source identity and citation
Markdown
Heading, explanation, code sample
Readable source content
Chunk
One coherent section
Retrieval unit
Embedding
Numeric vector
Semantic similarity search
Metadata
URL, title, section, version, hash
Filtering, citations, updates
Pinecone record
stable ID + values + metadata
Upsert and query target
Keep the original page text in metadata only when it fits your data model and privacy requirements. In this example, the chunk text is attached to each vector’s metadata so retrieval can return a citation-ready passage. For larger record designs, evaluate the current Pinecone data-modeling and upsert guidance.
Prerequisites: Crawl API, embeddings, and Pinecone
Create an Nstdata Crawl token, an OpenAI API key, and a Pinecone project. Create a dense Pinecone index whose dimension matches the embedding model output. The example below uses the default 1,536-dimensional output of text-embedding-3-small; if you set a custom embedding dimension, configure the index to the same value. Install the libraries and pin the exact versions resolved in your deployment lockfile:
The SDK names and API shapes below follow current vendor documentation. No API credentials are available in the writing environment, so the complete cloud workflow has not been run end to end.
Step 1: Crawl a site into Markdown
For a known URL, use the documented synchronous Python client. For a documentation site, submit a bounded site crawl with a low maxPages during the first validation run. The site-level API returns a task ID; poll status and then page results. The Crawl reference documents POST /api/v1/crawl, GET /api/v1/crawl/status/{crawlId}, and GET /api/v1/crawl/pages/{crawlId}.
import os
from nstdata_ai_crawl import CrawlRequestDto, Format, NstDataClient
with NstDataClient(os.environ["NSTDATA_API_TOKEN"])as client: submitted = client.submit_crawl_task(CrawlRequestDto( url="https://docs.example.com/", formats=[Format.MARKDOWN], maxDepth=2, maxPages=100, includeUrls=["*docs.example.com/*"], excludeUrls=["*docs.example.com/search*","*docs.example.com/login*"], ignoreQuery=True, onlyMainContent=True, timeout=60_000,)) crawl_id = submitted.idprint("submitted crawl", crawl_id)
The SDK example confirms the submission method and request fields. For production, add polling with a monotonic deadline, exponential backoff, and a maximum wait; then paginate the returned page list until its nextCursor is empty. A page record may carry a content reference rather than the full Markdown, so fetch the referenced artifact through the documented storage-read endpoint before chunking. Check each page’s success, status, metadata, and content rather than treating a completed parent crawl as proof every page is usable.
The Crawl guide’s reference response for large outputs uses fields such as markdownRef and an authenticated GET /api/v1/crawl/storage/read?st={ref}. Keep that retrieval step in the adapter for your SDK version; do not assume every response embeds a full body. Avoid crawling login-only or otherwise unauthorized content.
Step 2: Clean and normalize before embedding
Preserve heading paths, code fences, tables, and source links. Drop only repeated chrome that is clearly not part of the document. A basic normalization helper is deliberately conservative:
Do not remove every link or table just to reduce token count. API documentation often stores the useful answer in a parameter table or a linked version page. Capture the final canonical URL, title, language, and product version as metadata alongside the normalized body.
Step 3: Split into heading-aware chunks
A first version can split Markdown by headings while leaving fenced code together. This example creates one record for each heading section; add a tokenizer-aware second pass when a section exceeds your embedding provider’s input limit.
defsplit_sections(markdown:str): heading_path, lines, fenced =[],[],Falsefor line in markdown.splitlines():if line.strip().startswith("```"): fenced =not fenced
if line.startswith("#")andnot fenced and lines: text ="\n".join(lines).strip()if text:yield" / ".join(heading_path), text
level =len(line)-len(line.lstrip("#")) heading_path = heading_path[:level -1]+[line[level:].strip()] lines =[line]else: lines.append(line) text ="\n".join(lines).strip()if text:yield" / ".join(heading_path), text
This is a structure-first starter, not a universal chunker. It does not guarantee a maximum token count or merge tiny sections. The LangChain text-splitter guide provides another framework implementation to compare. Add recursive splitting for large sections and test boundary behavior on code blocks and tables. Track token size with the tokenizer associated with your embedding model rather than treating character count as an exact token count.
For the following sections, assume the page-results adapter has produced one validated record named page, with url, title, markdown, crawled_at, and content_hash fields. Materialize the splitter output once: sections = list(split_sections(page.markdown)). The previous site-crawl step submits a task; your adapter must poll it, read referenced Markdown artifacts, and construct this page record before embedding.
Step 4: Generate embeddings
The OpenAI embeddings guide documents the embeddings endpoint. Embed each chunk using one fixed model, and retain the model name with the index version so a later model change cannot silently mix incompatible vector spaces.
from openai import OpenAI
embedder = OpenAI()texts =[chunk_text for _, chunk_text in sections]response = embedder.embeddings.create( model="text-embedding-3-small",input=texts,)vectors =[item.embedding for item in response.data]assertlen(vectors)==len(texts)
Batching reduces request overhead, but size batches using provider limits and retry transient errors with backoff. Check vector length before writing:
expected_dimension =1536ifany(len(vector)!= expected_dimension for vector in vectors):raise ValueError("Embedding dimension does not match the Pinecone index")
Step 5: Create the Pinecone index
Create a dense index in the Pinecone console or with its current control-plane API. Set the index dimension to the embedding output dimension and choose the similarity metric required by your retrieval design. The host URL is needed for data-plane operations. Use a development index first so failed experiments do not contaminate production data.
Step 6: Upsert chunks with stable IDs
Pinecone’s current Python examples accept dense records with id, values, and optional metadata. Use a namespace to separate a documentation collection or tenant. A stable prefix makes it possible to delete all chunks for a changed source page before replacing them.
Upsert overwrites an existing record with the same ID. It does not automatically remove old chunk IDs that disappeared after a page edit. Maintain the previous page-to-ID list and delete old IDs after the replacement set has been validated. Pinecone documents batching and namespaces in its upsert guide.
Step 7: Query the website index
Embed the question with the same model and query Pinecone. Return metadata so the application can show a source link and heading instead of emitting a bare answer.
question ="How does the API authenticate requests?"query_vector = embedder.embeddings.create( model="text-embedding-3-small",input=question,).data[0].embedding
matches = index.query( vector=query_vector, top_k=5, include_metadata=True, namespace=namespace,)formatchin matches.matches:print(match.metadata["source_url"],match.metadata["heading_path"])print(match.metadata["text"])
Treat top_k as a tunable retrieval parameter, not a quality guarantee. Evaluate with a small set of real questions and check that each returned passage supports the answer. For access-controlled sites, also apply tenant and document permissions before returning context to the model.
Step 8: Handle updates and deletions
Store a hash of normalized page content. When the next accepted crawl returns the same hash, skip embedding. When the hash changes, create the new chunk list, upsert it, verify it is queryable, and then remove previous IDs. Keep a manifest outside Pinecone with canonical URL, content hash, crawl timestamp, model version, and chunk IDs. If a source crawl fails, retain the last known good index and flag the page for review; a timeout must not become an accidental delete.
Common errors and cost model
A dimension mismatch means the index and embedding output do not agree. A duplicate flood usually means IDs were random or included the ingestion timestamp. Missing citations usually means source_url was lost during chunking. A metadata query that returns no records may point to a namespace mismatch rather than an embedding problem.
Estimate cost per accepted page with separate terms: Crawl URL use, proxy traffic when enabled, embeddings for new or changed text, Pinecone writes and storage, and query-time model calls. A page can produce many chunks, so page volume alone is not a reliable storage estimate. See current Nstdata Crawl pricing and current Pinecone billing documentation when preparing a budget; rates and plan limits can change.
Pinecone versus other vector stores for this workflow
Pinecone is a managed choice when you want a hosted vector index and do not want to operate the database process. Chroma is convenient for local prototypes; Qdrant and Weaviate provide other managed or self-hosted deployments; pgvector can fit a team that already operates PostgreSQL. Compare filter needs, operations, backups, region, and permission model—not just vector-search latency. This tutorial uses Pinecone because the keyword targets that integration, not because one database is best for every workload.
A reliable website-to-vector database flow depends on the contracts between stages: Crawl produces a page artifact, your code validates and chunks it, the embedding model produces vectors of a known dimension, and Pinecone stores each vector under a stable ID with source metadata. Build the URL manifest and update/delete behavior before ingesting a large site. That small amount of bookkeeping prevents stale passages and broken citations later.
Discover and crawl the allowed pages, validate and chunk their text, generate embeddings, then upsert each vector with a stable ID and source metadata.
Q: Does Pinecone store the full webpage?
Pinecone stores vector records and associated metadata; your application decides whether chunk text is stored as metadata or in a separate content store.
Q: What is the best chunk size for Pinecone?
There is no universal best size; tune chunk boundaries and retrieval settings against representative questions and the embedding model’s input limits.
Q: How do I update website embeddings?
Compare a normalized content hash, re-embed changed pages, upsert replacement chunks, and remove the previous chunk IDs after validation.
Q: Can I store multiple websites in one index?
Yes, but use namespaces or metadata filters deliberately and enforce tenant permissions in your application before returning retrieved content.
Understand seven dimensions of RAG data quality and how to validate crawled pages, compare clean and raw inputs, preserve provenance, and diagnose retrieval failures. It separates extraction, retrieval, and generation errors so teams can debug the right stage.
Lena Zhou
Oct. 20th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
