What chunk size works best for RAG?

On our test of 94 passages and 25 questions, fixed 512 tokens with 64 tokens of overlap gave recall at 10 of 0.96 and nDCG at 10 of 0.796, while 256 tokens gave the best recall at 5 of 0.92. The spread across all six strategies we tested was only 0.035 nDCG, so 512 with overlap is a safe default, not a magic number. Measure on your own corpus before tuning further.

Does semantic chunking beat fixed size chunking?

Not reliably. In our run semantic chunking with a 0.45 similarity threshold shattered the corpus into 922 single sentence chunks, mean 33.9 tokens, took 56.1 seconds to build against 0.1 for fixed chunking, and scored the lowest recall at 5 at 0.84. A Vectara study at NAACL 2025 reached the same conclusion: the extra compute is not justified by consistent gains. Semantic chunking can win with a well chosen threshold, but a bad threshold is worse than fixed size.

How much overlap should RAG chunks have?

About 10 to 20 percent of the chunk size. We used 64 tokens of overlap on 512 token chunks, about 12 percent, and recall at 10 rose from 0.92 to 0.96. An independent study found overlap of 64 to 256 tokens moved recall by under 3 points while adding up to 96 percent more indexed text, so more overlap is not free. Keep it small and measure.

What is contextual retrieval and how much does it help?

Anthropic's September 2024 method: a small model writes 50 to 100 tokens of context situating each chunk in its whole document, prepended before embedding and before BM25 indexing. Retrieval failure in the top 20 fell from 5.7 percent to 3.7 percent with contextual embeddings alone, 2.9 percent with contextual BM25, and 1.9 percent with reranking, a 67 percent total cut. Cost is about $1.02 per million document tokens with prompt caching.

What is late chunking?

From Jina AI: run the embedding transformer over the whole document first, then average the token vectors inside each chunk boundary, so each chunk embedding keeps information from surrounding text. On BEIR with 256 token chunks it beat naive chunking on NFCorpus 29.98 against 23.46 nDCG at 10, SciFact 66.10 against 64.20, and TRECCOVID 64.70 against 63.36. It needs a long context embedding model and does not help on every dataset.

When should teams use LangChain's RecursiveCharacterTextSplitter?

Yes for structured documents, and it is a fine default for plain text too. It tries separators in order, paragraph, line, sentence, word, character, with example defaults of 1,000 characters and 200 overlap. On our corpus it scored recall at 10 of 0.96 but the lowest nDCG at 0.770, slightly behind fixed 512. Use it when your documents have headers, code, or lists; for plain prose it performs about the same as fixed size chunking.

RAG Deep: Chunking Strategies Tested at Scale