Also known as: passage segmentation, document chunking
Chunking is the process of splitting long documents into smaller passages that fit cleanly inside an embedding model's context window — and that align with semantic boundaries so each chunk is independently retrievable.
Chunking splits a long document — a 50-page contract, a 200-page manual, a 30-minute meeting transcript — into pieces small enough that an embedding model can encode each one and a retrieval system can return individual chunks. The choices you make here cascade through every later stage of retrieval quality; bad chunking will ruin a state-of-the-art embedding model.
Embedding models have a fixed context length — 8K, 32K, 128K tokens depending on the model. Beyond it you either truncate (losing tail content) or chunk. Even when documents fit, embedding a whole document into one vector dilutes the signal: the relevant span gets averaged with everything else. Passage-sized chunks win on precision.
Embedding models are trained on a particular distribution of input lengths — typically passage-shaped chunks of 100-500 tokens. If you embed a 50-token snippet, you’re operating below the training distribution; the embedding under-uses the model’s representational capacity and the resulting vector is noisier than it should be. Embed a 4000-token document into a model trained on 256-token passages and you get the opposite failure: the embedding is the average of many topically-distinct spans, and the cosine similarity to any specific query degrades because the relevant span is diluted.
The practical consequence is that swapping embedding models almost always means re-tuning chunk size. A model trained for long passages (Cohere embed-v3, gte-large with extended context) tolerates 1024-token chunks fine; a model trained for retrieval over Wikipedia-shaped passages (e5-small, bge-small) wants 256-512. The instinct to “set chunk size once and forget it” is exactly wrong — chunk size is a hyperparameter of the chunker-plus-embedder pair, not the chunker alone.
A second-order effect: chunk length distribution should be relatively uniform. Mix 50-token and 2000-token chunks in the same index and the cosine-similarity comparisons across them are no longer apples-to-apples — the long chunks tend to have lower-magnitude vectors after normalization simply because they’re embedding more topics at once.
256-512 tokens with ~15% overlap is a safe default for most prose. Push larger (768-1024) for technical reference docs where context per chunk matters; smaller (128-256) for FAQ-shaped content where each chunk is a self-contained answer.
Prepend a breadcrumb — document title plus heading hierarchy — to every chunk before embedding. The bi-encoder gets the global anchor for free, and the reranker gets a much cleaner signal because chunks like 'this provision applies' aren't context-free anymore.
Yes. Even with a 32K context window, encoding a whole document into one vector dilutes the relevant span — the embedding becomes the average of everything in the doc, which is rarely what you're searching for. Passage-level chunks still win on precision.
