For web-crawled RAG, chunk boundaries should follow the document’s meaning and structure before they follow a token limit.
Markdown headings work well for docs; recursive splitting is a practical fallback for long sections; semantic and parent-child approaches need their own evaluation.
Keep source URL, heading path, version, and stable IDs with each chunk, and protect code fences, tables, and endpoint/parameter context.
Compare strategies on the same pages and question set; this draft gives an evaluation method, not fabricated benchmark results.
## Introduction: Chunking is part of the retrieval design
Nstdata Crawl is the collection layer in this workflow; its [product overview](https://www.nstdata.io/scraping) describes page and site retrieval for downstream data pipelines.
Chunking divides source documents into units that can be embedded and retrieved. The goal is not to produce equal-sized text blocks. It is to make it likely that a passage contains enough context to answer a question without dragging in unrelated material. A web page with a heading, an API table, a code sample, and a footer is not equivalent to an unstructured paragraph.
A good RAG chunk preserves the information needed to interpret the passage: its heading, page, version, and often the sentence or code block beside it. A poor chunk can split an endpoint from its parameters or repeat navigation text in every result. The method below prepares clean Markdown first, then compares chunking choices on the same retrieval task.
What is chunking in RAG and why does it matter?
The retriever searches chunks, not the whole source site. If one chunk is too broad, a relevant sentence competes with unrelated content and consumes more context. If one chunk is too narrow, a retrieved term may be separated from its definition, condition, or example. This can lower retrieval usefulness even when the source page was captured correctly.
Chunking cannot repair missing content. If a crawler failed to capture the page’s client-rendered table, no splitter will reconstruct it. Use Crawl to retrieve and clean the page, then inspect the Markdown structure before tuning a chunker. Nstdata’s Crawl guide documents Markdown output for RAG and agent workflows.
Web content includes repeated headers and footers, nested headings, code samples, tables, links, notices, and versioned paths. A character-only splitter has no idea that a code fence is one example or that the heading “Authentication” defines the scope of the next paragraphs. Normalize boilerplate first, then preserve the page structure that carries meaning.
Six common RAG chunking strategies
Strategy
Boundary
Good fit
Cost or failure mode
Fixed-size
characters or tokens
quick baseline
splits headings, examples, and lists
Recursive
paragraphs, lines, sentences
mixed prose
separator order and overlap need tuning
Semantic
meaning change between sentences
prose with topic shifts
extra embedding work; threshold-sensitive
Markdown heading
H1/H2/H3 sections
technical docs and knowledge bases
very long sections need a second pass
Parent-child
small child, larger parent
precise retrieval plus fuller context
more IDs and retrieval logic
Contextual/hierarchical
section plus document context
fragmented pages or multi-level docs
added tokens and possible context injection
These are implementation patterns, not a ranking. The right choice depends on the questions your system must answer and the cost of missing context.
Prepare Markdown before you chunk it
A useful preprocessing order is: reject empty or failed pages, remove duplicate navigation that survived extraction, normalize whitespace, preserve title/heading hierarchy, then split. Keep code fences, tables, links, warning blocks, and method/endpoint pairs intact. The current Nstdata output supports Markdown and cleaned HTML; Markdown is a practical starting format for text retrieval, while HTML can be useful if custom parsing is needed.
Avoid dropping a section simply because it contains short text. A short “Deprecated” warning may be more important than a long overview. Add validation to detect empty bodies, repeated content hashes, unclosed code fences, and very long sections before they reach embeddings.
Step by step: implement three strategies in Python
This stdlib-only example provides fixed-size, recursive, and heading-aware baselines. It is meant to compare boundaries on a sample Markdown fixture; add your production tokenizer and test suite before indexing real content.
import re
deffixed_chunks(text:str, size:int=1200, overlap:int=150):if size <=0ornot0<= overlap < size:raise ValueError("require 0 <= overlap < size") start =0while start <len(text): end =min(start + size,len(text))yield text[start:end]if end ==len(text):break start = end - overlap
defrecursive_chunks(text:str, size:int=1200): parts = re.split(r"\n\n+|(?<=[.!?])\s+", text) chunks, current =[],""for part in parts: candidate =f"{current}\n\n{part}"if current else part
iflen(candidate)> size and current: chunks.append(current) current = part
else: current = candidate
if current: chunks.append(current)return chunks
defheading_chunks(markdown:str): path, lines, fenced =[],[],Falsefor line in markdown.splitlines():if line.strip().startswith("```"): fenced =not fenced
if line.startswith("#")andnot fenced and lines: body ="\n".join(lines).strip()if body:yield" / ".join(path), body
level =len(line)-len(line.lstrip("#")) path = path[:level -1]+[line[level:].strip()] lines =[line]else: lines.append(line)if lines:yield" / ".join(path),"\n".join(lines).strip()
The fixed-size version measures characters, not model tokens. Recursive splitting uses paragraph and sentence boundaries but can still split a long table. Heading splitting preserves fenced blocks but may emit a section that exceeds your model’s input limit. These are baselines you can run on a fixture, not drop-in guarantees for every Markdown dialect.
Compare chunk boundaries on the same document
Use one representative page that contains prose, a multi-row table, a multi-line code fence, and at least three heading levels. For each strategy, record chunk count, approximate and actual token distribution, number of code fences preserved, number of chunks with heading context, and duplicate rate. Inspect the outliers instead of reporting only averages.
A comparison worksheet might look like this:
Strategy
Chunk count
Median / max tokens
Code fences intact
Citation metadata present
Fixed-size
fill from run
fill from tokenizer
manually count
yes/no
Recursive
fill from run
fill from tokenizer
manually count
yes/no
Heading-first
fill from run
fill from tokenizer
manually count
yes/no
Do not fill this table with invented benchmark numbers. Run it on your target corpus and attach the script version and tokenizer to the result.
How to choose chunk size and overlap
Chunk size interacts with query shape, the embedding model’s input limit, top-k, reranking, and generation context. Overlap can preserve a boundary sentence, but it also creates repeated text, duplicate search results, extra embedding cost, and less result diversity. Begin with heading-aware chunks; add a second split only for sections that exceed the model limit. Compare a small grid of sizes and overlaps rather than copying a “best” number from another dataset.
Useful questions include: “What header authenticates a call?”, “Which parameter accepts a list?”, “What does the error response mean?”, and a question spanning two headings. Track Recall@K or hit rate against labelled evidence, MRR or rank, answer correctness, citation correctness, and context tokens. Keep retrieval failures separate from generation failures: a model cannot cite a relevant passage that never reached top-k.
Handle code, tables, FAQs, and hierarchical context
For code, preserve the fenced block and its language label. If it is too long, split at function or example boundaries and attach the nearest heading plus file or language name. For tables, repeat the header row in each split and keep a row’s label with its value. FAQ entries can often be one question-answer pair per chunk. Product specifications should keep units and field names attached to values.
Parent-child retrieval uses a small child chunk for matching and retrieves a larger parent section for generation. Store parent_id, child_id, source_url, and heading_path. Contextual chunking can add the document title or section path to the text before embedding, but test whether that extra context biases similarity or duplicates a generic title across many pages.
Build a RAG evaluation question set
Create questions from actual support tickets and developer tasks, then label the exact source passage that answers each one. Include direct facts, table lookups, code questions, version-specific questions, cross-section questions, and unanswerable questions. Have a reviewer verify the answer and its evidence rather than treating another LLM’s label as ground truth.
Measure retrieval and answer quality separately. For retrieval, record whether a labelled chunk appears in top-k and at what rank. For generation, record correctness, source link correctness, unsupported claims, and whether the system abstains when the page does not answer the question. Repeat the experiment after a document update so freshness regressions are visible.
Common chunking mistakes
Blind fixed windows: meaningful headings and examples are split without metadata.
Too much overlap: near-duplicate chunks dominate top-k and cost more to embed.
Lost title paths: a definition appears without the product or API it describes.
Truncated code or tables: the retrieval result no longer represents valid instructions.
Mixed versions: v1 and v2 API guidance are embedded without a filterable version field.
No evaluation set: a chunker is selected by appearance rather than retrieval performance.
Choose by use case
For developer documentation, start with Markdown headings and split unusually long sections recursively. For FAQs, keep question-answer pairs intact. For product knowledge, preserve attribute-value rows and version/status metadata. For long reports, try parent-child retrieval or semantic segmentation if the question set crosses sections. For news, keep each item’s date and publisher attached and consider short-lived records. Compare each with the same labelled query set. For implementation details, see the LangChain splitter concepts, OpenAI embeddings guide, and Pinecone semantic-search guide. These sources document APIs and patterns; they do not establish which strategy wins on your corpus.
Final checklist
Clean and validate source content before chunking.
Preserve title path, source URL, version, language, and update time.
Keep code fences, tables, and tightly coupled field definitions intact.
Use a second splitting pass only where the model input limit requires one.
Test retrieval and answer quality on the same fixed questions.
Record actual token and vector costs for the chosen parameters.
There is no universal “best chunking strategy for RAG.” For web-crawled documentation, structure-first splitting is a useful baseline because the page already contains headings and code boundaries. Recursive or semantic strategies can improve specific corpora, but only an evaluation set can show whether they help your users’ questions. Keep the comparison reproducible and record both retrieval quality and operational cost.
The best strategy is the one that retrieves complete evidence for your real queries at an acceptable token and storage cost; heading-aware splitting is a practical baseline for structured web docs.
Q: What is the ideal chunk size?
There is no universal ideal; evaluate sizes against the embedding model’s input limit, answer context, and labelled retrieval questions.
Q: Does chunk overlap improve retrieval?
Overlap can preserve context near boundaries, but too much overlap creates duplicate results and extra indexing cost, so test a small range.
Q: What is semantic chunking?
Semantic chunking uses changes in meaning or similarity between passages to propose boundaries, which may help prose but introduces model and threshold choices.
Q: How should I chunk Markdown for RAG?
Start with heading sections, carry the heading path as metadata, preserve fenced code and table headers, and recursively split only long sections.
Understand seven dimensions of RAG data quality and how to validate crawled pages, compare clean and raw inputs, preserve provenance, and diagnose retrieval failures. It separates extraction, retrieval, and generation errors so teams can debug the right stage.
Lena Zhou
Oct. 20th 2026
Crawl entire websites with a single API request
99.8% success rate with JavaScript rendering
Get clean, LLM-ready data in multiple formats
Turn any website into Markdown, HTML, JSON, links, PDFs and more — without managing crawling infrastructure.
