RAG Chunking Strategies: How to Test Their Impact on Quality
Chunking is one of the most consequential decisions in any RAG system, and one of the least tested. Most teams pick a strategy, set a chunk size, and ship. Then they wonder why retrieval quality degrades on longer documents, or why the system can never answer questions that span multiple sections.
The problem is not that teams make the wrong choice — it is that they have no framework for measuring whether their choice is correct. This guide gives you that framework.
Why Chunking Matters More Than You Think
When a user asks a question, your vector database searches for chunks similar to the query — not full documents. The chunk is the atomic unit of retrieval. If the answer to a question spans a chunk boundary, neither chunk may score high enough to be retrieved. If chunks are too large, they dilute the signal and cause lower similarity scores across the board. If they are too small, answers get split across many chunks and the context becomes fragmented.
The chunking strategy you choose affects every downstream metric: retrieval recall, context noise ratio, answer faithfulness, and ultimately user satisfaction.
The Four Main Strategies
1. Fixed-Size Chunking
Split text into fixed-length chunks of N tokens, with optional overlap:
from langchain.text_splitter import TokenTextSplitter
def fixed_size_chunker(text: str, chunk_size: int = 512, overlap: int = 50) -> list[str]:
splitter = TokenTextSplitter(
chunk_size=chunk_size,
chunk_overlap=overlap
)
return splitter.split_text(text)When it works well: Homogeneous documents (news articles, forum posts, product descriptions) where content density is relatively uniform.
When it breaks: Technical documentation, legal contracts, anything with section structure. A 512-token chunk might cut a code example in half, or include the end of one section and the start of another, creating noise.
2. Recursive Character Splitting
Split on natural boundaries (paragraphs, sentences, words) falling back to smaller units when needed:
from langchain.text_splitter import RecursiveCharacterTextSplitter
def recursive_chunker(text: str, chunk_size: int = 1000, overlap: int = 100) -> list[str]:
splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=overlap,
separators=["\n\n", "\n", ". ", " ", ""]
)
return splitter.split_text(text)This is the default in LangChain and works well for most prose. It respects natural text boundaries while still enforcing a size limit.
3. Semantic Chunking
Group sentences by semantic similarity — when the topic shifts, start a new chunk:
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
def semantic_chunker(text: str, threshold: float = 0.5, model_name: str = "all-MiniLM-L6-v2") -> list[str]:
model = SentenceTransformer(model_name)
sentences = text.split(". ")
embeddings = model.encode(sentences)
chunks = []
current_chunk = [sentences[0]]
for i in range(1, len(sentences)):
similarity = cosine_similarity(
embeddings[i-1].reshape(1, -1),
embeddings[i].reshape(1, -1)
)[0][0]
if similarity < threshold:
chunks.append(". ".join(current_chunk))
current_chunk = [sentences[i]]
else:
current_chunk.append(sentences[i])
if current_chunk:
chunks.append(". ".join(current_chunk))
return chunksWhen it works well: Documents with distinct topic sections, long-form content, mixed-topic documents.
When it breaks: Dense technical text where every sentence is about the same narrow topic — you can get enormous single chunks. Also slower and more compute-intensive than other strategies.
4. Sliding Window Chunking
Overlapping windows that ensure no information falls through the cracks:
def sliding_window_chunker(
text: str,
window_size: int = 400,
step_size: int = 200
) -> list[str]:
tokens = text.split() # Simplified; use proper tokenizer in production
chunks = []
for i in range(0, len(tokens) - window_size + 1, step_size):
chunk = " ".join(tokens[i:i + window_size])
chunks.append(chunk)
return chunksWhen it works well: Questions that might span section boundaries, ensuring high recall at the cost of more redundant chunks.
When it breaks: Creates many duplicate or near-duplicate chunks, which can confuse the retriever and inflate your vector store.
Building a Chunk Quality Benchmark
To compare strategies objectively, you need a benchmark. Here is a complete framework:
Step 1: Prepare Your Test Corpus
Gather a representative sample of your actual documents — at least 20-30 documents covering different types (short, long, dense technical, narrative prose). Do not use toy examples; test on real content.
import json
from pathlib import Path
# Load your test corpus
test_corpus = []
for path in Path("test_documents/").glob("*.txt"):
test_corpus.append({
"id": path.stem,
"content": path.read_text(),
"doc_type": classify_document(path.stem) # Your classification logic
})Step 2: Create a Question-Answer-Document Golden Dataset
For each test document (or a representative sample), create questions where you know which chunk should be retrieved:
GOLDEN_DATASET = [
{
"doc_id": "product-manual-v2",
"question": "What is the default timeout for API calls?",
"answer": "30 seconds",
"answer_location": "section_3_2", # Human-labeled section
"answer_start_char": 4821, # Character offset in source doc
"answer_end_char": 4890,
},
{
"doc_id": "legal-terms-2024",
"question": "Can users export their data after account deletion?",
"answer": "Within 30 days of account deletion",
"answer_location": "section_12",
"answer_start_char": 18234,
"answer_end_char": 18312,
}
]Step 3: Chunk Overlap Score
The most important metric for a chunking strategy is whether the answer lands fully within a single chunk. Compute the answer containment rate:
def answer_containment_rate(
chunks: list[str],
source_doc: str,
golden_cases: list[dict]
) -> dict:
"""
For each golden case, check if the answer text appears fully in at least one chunk.
Returns containment rate and cases where answer was split.
"""
results = {
"total": len(golden_cases),
"contained": 0,
"split": 0,
"split_cases": []
}
for case in golden_cases:
answer = case["answer"].lower()
found_in_chunk = any(answer in chunk.lower() for chunk in chunks)
if found_in_chunk:
results["contained"] += 1
else:
results["split"] += 1
results["split_cases"].append({
"question": case["question"],
"answer": case["answer"]
})
results["containment_rate"] = results["contained"] / results["total"]
return resultsStep 4: Chunk Size Distribution
Healthy chunking produces a predictable size distribution. Runaway large chunks or many tiny useless chunks both indicate problems:
import statistics
def analyze_chunk_distribution(chunks: list[str]) -> dict:
sizes = [len(chunk.split()) for chunk in chunks]
return {
"count": len(chunks),
"mean_size": statistics.mean(sizes),
"median_size": statistics.median(sizes),
"std_dev": statistics.stdev(sizes) if len(sizes) > 1 else 0,
"min_size": min(sizes),
"max_size": max(sizes),
"chunks_under_50_words": sum(1 for s in sizes if s < 50),
"chunks_over_1000_words": sum(1 for s in sizes if s > 1000),
}Step 5: End-to-End Retrieval Recall
Plug each chunking strategy into your actual retriever and measure recall against your golden dataset. This is the ground truth metric — everything else is diagnostic:
import pytest
from your_vector_store import VectorStore
from your_embedder import Embedder
STRATEGIES = {
"fixed_512": lambda doc: fixed_size_chunker(doc, chunk_size=512, overlap=50),
"fixed_256": lambda doc: fixed_size_chunker(doc, chunk_size=256, overlap=25),
"recursive_1000": lambda doc: recursive_chunker(doc, chunk_size=1000, overlap=100),
"semantic_05": lambda doc: semantic_chunker(doc, threshold=0.5),
"sliding_400": lambda doc: sliding_window_chunker(doc, window_size=400, step_size=200),
}
@pytest.mark.parametrize("strategy_name,chunker", STRATEGIES.items())
def test_retrieval_recall_by_strategy(strategy_name, chunker, test_corpus, golden_dataset):
embedder = Embedder()
# Build a fresh vector store for this strategy
store = VectorStore(namespace=f"test_{strategy_name}")
for doc in test_corpus:
chunks = chunker(doc["content"])
for i, chunk in enumerate(chunks):
embedding = embedder.embed(chunk)
store.upsert(
id=f"{doc['id']}_{i}",
embedding=embedding,
metadata={"doc_id": doc["id"], "chunk_index": i, "text": chunk}
)
# Measure recall
recalls = []
for case in golden_dataset:
query_embedding = embedder.embed(case["question"])
results = store.search(query_embedding, k=5)
retrieved_texts = [r.metadata["text"].lower() for r in results]
answer_found = any(case["answer"].lower() in text for text in retrieved_texts)
recalls.append(1.0 if answer_found else 0.0)
recall = sum(recalls) / len(recalls)
print(f"\nStrategy {strategy_name}: recall@5 = {recall:.3f}")
# Assert minimum acceptable recall
assert recall >= 0.70, f"Strategy {strategy_name} recall too low: {recall:.3f}"
# Clean up
store.delete_namespace(f"test_{strategy_name}")Measuring Impact on Retrieval Recall
When you run the benchmark above, you will typically see patterns like these (actual numbers vary by domain):
| Strategy | Answer Containment | Recall@5 | Avg Chunk Size | Notes |
|---|---|---|---|---|
| fixed_256 | 71% | 0.68 | 256 tokens | Splits answers frequently |
| fixed_512 | 82% | 0.77 | 512 tokens | Better but still misses cross-boundary answers |
| recursive_1000 | 89% | 0.83 | ~600 tokens | Good default for most prose |
| semantic_0.5 | 91% | 0.86 | Variable (100-800) | Best containment, slower indexing |
| sliding_400 | 95% | 0.89 | 400 tokens | Highest recall, 2x chunk count, more noise |
The sliding window consistently achieves the highest recall because overlap ensures no answer falls through a boundary. But it creates a noisier context — you retrieve more semi-relevant chunks alongside the relevant ones. Test context noise ratio separately.
Testing Chunk Quality Regressions
Once you have selected a strategy, lock it down with regression tests. Changes to chunk size, overlap, or separator characters can silently degrade quality:
# tests/test_chunking_regression.py
import pytest
from your_chunkers import recursive_chunker
REGRESSION_BASELINES = {
"product_docs": {
"recall_at_5": 0.83,
"containment_rate": 0.89,
"mean_chunk_size": 612,
}
}
def test_chunking_regression_product_docs(product_docs_corpus, product_docs_golden):
all_chunks = []
for doc in product_docs_corpus:
chunks = recursive_chunker(doc["content"], chunk_size=1000, overlap=100)
all_chunks.extend([(doc["id"], i, c) for i, c in enumerate(chunks)])
containment = answer_containment_rate(
[c for _, _, c in all_chunks],
product_docs_corpus,
product_docs_golden
)
baseline = REGRESSION_BASELINES["product_docs"]
assert containment["containment_rate"] >= baseline["containment_rate"] - 0.03, (
f"Containment rate regressed: {containment['containment_rate']:.3f} "
f"vs baseline {baseline['containment_rate']:.3f}"
)Run this in CI whenever anyone changes the chunking configuration. A 3-point drop in containment rate is a red flag.
Domain-Specific Considerations
Code documentation: Use a code-aware splitter that keeps functions and classes intact. LangChain's Language.PYTHON splitter does this. Never split on character count alone when chunking code.
from langchain.text_splitter import RecursiveCharacterTextSplitter, Language
code_splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.PYTHON,
chunk_size=1000,
chunk_overlap=100
)Legal and regulatory documents: Semantic chunking works best — legal sections are semantically distinct. Test that contract clauses never get split across chunks; a partial clause retrieved without its condition is dangerous.
Product manuals with numbered steps: Recursive splitting with "\n\n" and "\n" as primary separators usually keeps numbered lists intact. Verify by checking that no chunk contains step 3 but not step 2 for a sequential process.
Practical Recommendations
Start with recursive character splitting at 1000 tokens with 100 token overlap. This is the safe default that works reasonably well across most document types without extra compute overhead.
Then measure. Run your benchmark. If answer containment is below 85%, try semantic chunking or increase overlap. If you are seeing context noise (many irrelevant chunks retrieved), reduce chunk size or add a reranking step rather than changing the chunking strategy.
Never change chunking configuration without running the full benchmark. What feels like a minor tuning can silently drop recall by 10+ points on specific document types.
HelpMeTest can help you automate the end-user validation side of this — running regression tests against your live RAG application after each knowledge base or configuration change, verifying that answers your users rely on are still correct.
Summary
The right chunking strategy depends on your document corpus and query patterns. The process is:
- Build a golden dataset with questions and expected answer locations
- Compute answer containment rate for each strategy candidate
- Measure end-to-end retrieval recall@k in your actual vector store
- Lock in the winner with regression tests
- Run regressions in CI on every configuration change
Chunking is not set-and-forget. As your document corpus grows and your query patterns shift, revisit the benchmark. A strategy that worked well at launch may not be optimal six months later.
