RAG

How it works

First, the reference documents are split into small pieces of a few hundred tokens each (chunking). Each piece is converted into an embedding, a list of numbers that captures its meaning, and stored in a vector DB. When a user asks a question, the pieces closest in meaning to the question are found and pasted into the prompt. The large language model (LLM) then reads this material and produces an answer. In other words, it is a structure that ties 'finding (retrieval)' and 'writing (generation)' into one flow.

How it has evolved

In 2020, the REALM paper from Google researchers had a language model search and consult large document collections such as Wikipedia starting from the pre-training stage. It was the first to show that a 'retriever' could be trained jointly without labeled answers. In the same year, a paper by Patrick Lewis and colleagues gave the approach the name 'retrieval-augmented generation (RAG)'. That paper presented a general-purpose training method combining a Wikipedia retriever with a text generation model, and reported that it produced more specific and factual text than a generation model alone. A 2023 survey paper by Yunfan Gao and colleagues organized later developments into simple 'Naive RAG', 'Advanced RAG' with improved retrieval quality, and 'Modular RAG' that freely combines components.

Why it matters

LLMs do not know information from after their training ended, and they sometimes hallucinate, making up plausible-sounding content about things they do not know. With RAG, knowledge can be kept up to date by swapping in documents without retraining the model, which keeps costs low. It can also use non-public material such as company policies or product manuals. And because it can show which documents it consulted alongside the answer, users can easily check for themselves.

Things to keep in mind (limitations)

RAG reduces hallucinations but does not eliminate them. Even when the right material is retrieved, the model can produce a wrong answer if it misreads the context. The 2023 'Lost in the Middle' paper reported that models make less effective use of relevant information when it sits in the middle of a long input than when it is at the beginning or end. So attaching more search results is not always better. Splitting documents into small pieces can also strip away surrounding context, making it hard to find the needed piece, so complementary techniques such as rerankers and hybrid search are used alongside.

Example

Chatbots that answer questions about internal company policies and customer support bots that answer based on product manuals are typical RAG use cases. In September 2024, Anthropic introduced a method called 'Contextual Retrieval'. When documents are split, each piece gets a short description of its context in the original document to reduce retrieval failures; the company said this cut retrieval failures by 49%, and by 67% when combined with reranking. In the same post, the company also explained that if the reference material is smaller than about 200,000 tokens (around 500 pages), it can be simpler to put the entire material into the prompt without RAG.

Related terms

Embedding Vector DB Chunking Reranker Hybrid search Hallucination

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
• REALM: Retrieval-Augmented Language Model Pre-Training (2020)
• Retrieval-Augmented Generation for Large Language Models: A Survey (2023)
• Lost in the Middle: How Language Models Use Long Contexts (2023)
• Retrieval-augmented generation - Wikipedia
• Contextual Retrieval in AI Systems (Anthropic)