Chunking

How it works

First, the document is split into units such as a fixed length, paragraphs or sentences. Each piece is converted into a list of numbers called an 'embedding.' An embedding represents the meaning of text as numbers. The converted pieces are stored in a vector DB. When a user asks a question, the pieces closest in meaning to the question are found and passed to the AI, which uses them as reference to answer. According to Anthropic, a single chunk is usually no longer than a few hundred tokens.

Why it matters

Pinecone gives two reasons why chunking is needed. First, there is a limit to how much text an embedding model can process at once (its context window). Anything beyond this limit is cut off, and important information can be lost. Second, to be retrieved, each piece must contain useful information. Pieces made up only of sentences that do not make sense without the surrounding context are rarely matched in search. That is why how the pieces are split has a large effect on the quality of RAG answers.

How it has evolved

Early on, it was common to simply cut documents into pieces of a fixed length. The 2023 paper 'Dense X Retrieval' proposed using 'propositions' as the retrieval unit instead of paragraphs or sentences. A proposition is a short sentence that contains a single fact and makes sense on its own. The paper reported that splitting text this finely improved retrieval performance compared with paragraph-level units. The 2024 paper 'RAPTOR' looked at the opposite problem: short chunks alone make it hard to understand a document as a whole. It repeatedly grouped and summarized chunks to build a tree-shaped structure, so retrieval could happen at several levels, from fine details to the main storyline. In September 2024, Anthropic introduced a method called 'Contextual Retrieval'. It adds a sentence to each chunk explaining where that chunk fits in the whole document before storing it, and Anthropic said this reduces retrieval failures.

Things to know

There is no single right chunk size for every case. Large chunks keep context but make retrieval less precise, while small chunks are more precise but tend to lose context. So the usual practice is to test several approaches and choose one based on the type of documents and the kinds of questions. Anthropic also notes that if all the material is small enough to fit in the model's context window, it can be simpler to put the whole thing into the prompt without chunking or RAG.

Example

Suppose you are building a support chatbot that answers questions based on company policies. If you split a policy handbook hundreds of pages long into chunks by clause or paragraph and store them in a vector DB, then when an employee asks, "What's the deadline for submitting travel expenses?", you can find only the chunk containing the relevant clause and pass it to the AI. Retrieval works better if each chunk also notes where it belongs, such as 'part of Chapter 3 of the travel expense policy'. Anthropic's Contextual Retrieval comes from exactly this idea.

Related terms

RAG Embedding Vector DB Semantic search Context window Reranker

References

• Dense X Retrieval: What Retrieval Granularity Should We Use? (2023)
• RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (2024)
• Chunking Strategies for LLM Applications | Pinecone
• Contextual Retrieval in AI Systems (Anthropic)