Retrieval-augmented generation (RAG): giving a model an 'open book'

Why it was needed: knowledge in the head is not enough

Large language models (LLMs) remember a large share of the facts they saw in training data. But that knowledge is scattered across billions of parameters. The REALM paper pointed out that with this approach, storing more facts requires making the network ever larger.

Lewis et al. (2020), who first proposed RAG, named the drawbacks more specifically. Knowledge held in parameters cannot easily be expanded or revised, it is hard to show evidence for why an answer was given, and the model produces hallucinations, making up plausible but untrue content. On knowledge-intensive tasks, performance also lagged behind systems designed specifically for those tasks.

The 2023 RAG survey (Gao et al.) also listed hallucination, outdated knowledge and untraceable reasoning as challenges for LLMs. The core problem is that the world keeps changing while the model's knowledge stops at the point training ended.

Core idea: from a closed-book exam to an open-book exam

The idea behind RAG is 'instead of making the model memorize everything, let it open the book when needed.' A student in a closed-book exam writes a wrong answer if their memory is wrong. A student in an open-book exam finds and reads the relevant pages and writes an answer based on them.

Lewis et al. described this in terms of two kinds of memory. One is 'parametric memory' held in the model parameters; the other is 'non-parametric memory,' an external document store. In their RAG, a pre-trained generative model (BART) served as parametric memory, and an index of Wikipedia documents converted into vectors served as non-parametric memory.

The benefits of this split are clear. The book (the document store) can be edited and expanded directly, and you can inspect which pages the model consulted. In fact, the paper showed that the model's knowledge could be updated simply by replacing the document store.

How it works: preparation, retrieval, generation

First, there is a preparation step done in advance. Documents are split into suitably sized pieces (chunking), and each piece is converted into a numeric vector that captures its meaning (embedding). These vectors are stored in an index such as a vector DB so they can be searched quickly. In Lewis et al.'s RAG, Wikipedia was built into such a 'dense vector index' ahead of time.

Second, when a question comes in, the retriever converts the question into a vector as well and finds the top K documents in the index closest to the question vector. The advantage of semantic search is that it can find documents with similar meaning even when the words do not match exactly. The original paper used a pre-trained neural retriever called DPR for this.

Third, the generative model takes the original question and the retrieved documents together as input and produces an answer. Lewis et al. compared two methods here. RAG-Sequence generates the whole answer relying on the same single document, while RAG-Token can consult a different document for each word (token), so it can combine content from multiple documents. They also fine-tuned the retriever and generator together so the two components adapt to each other.

RAG used in practice today usually keeps the retriever and the LLM separate and pastes documents into the prompt, but the backbone of 'preparation → retrieval → generation with documents' is the same.

How it developed: from REALM to agentic RAG

REALM (February 2020) attached a knowledge retriever to the language model from the pre-training stage. Using fill-in-the-blank (masked language modeling) as the training signal, it also trained the retrieval step over millions of documents. The paper stated it was the first to pre-train a retriever without labels (in an unsupervised way), and on three open-domain question-answering benchmarks its accuracy was 4–16 percentage points higher than previous methods.

RAG (May 2020) extended this hybrid memory to generative models that 'produce sentences.' It aimed to go beyond REALM, which handled only extractive question answering that pulls answers out of documents. RAG achieved state-of-the-art results at the time on three open-domain question-answering tasks, and on generation tasks it produced more specific, diverse and factual text than a BART-only model.

Self-RAG (2023) took up the question of 'when to retrieve.' Its point was that always attaching a fixed number of documents, whether needed or not, can make answers worse because of irrelevant documents. Self-RAG trained the model to generate special tokens called 'reflection tokens,' so it decides for itself whether retrieval is needed and evaluates both the relevance of retrieved documents and whether its own answer is supported by them. With 7B and 13B models, it reported better results than ChatGPT and retrieval-augmented Llama2-chat on question answering, reasoning and fact verification, and it also improved citation accuracy.

The same year, the survey by Gao et al. organized this progression into three stages: Naive RAG (simple retrieve-then-generate), Advanced RAG (improvements before and after retrieval), and Modular RAG (assembling components like building blocks). The 2025 agentic RAG survey argued that fixed workflows struggle with multi-step reasoning and classified architectures in which AI agents dynamically manage retrieval strategy through planning, tool use, reflection and multi-agent collaboration.

Where it is heading now: robustness and reliability

One strand of recent research asks, 'What if the knowledge base is not clean?' RAGStress (2026) pointed out that RAG evaluations usually assume a clean knowledge base. It tested four kinds of contamination (factual distortion, numeric typos, relevance pollution and contradiction injection) at three levels of intensity. It found that clean retrieval settings hide differences in robustness between systems, that contamination that changes meaning itself is more harmful, and that models that answer well without retrieval are not necessarily robust to contaminated retrieval.

Another strand is the problem that 'retrieval does not guarantee the answer is factual.' A conformal factuality control study (2026) applied a statistical filter, which splits an answer into individual claims and keeps only those with verified evidence, to multi-step retrieval (multi-hop) RAG. At a 95% target, the share of responses in which all remaining claims were supported was 95.80–97.20%, a large increase from without the filter (55.60–76.03%). But only 4.41–31.09% of claims remained, and many answers ended up empty. Reliability and the amount of answer trade off against each other.

Real-world builds are also moving toward layering multiple techniques. An open-source RAG pipeline (2026) covering about 230,000 papers in high-energy and astroparticle physics gathers candidates with hybrid search, which uses keyword matching and semantic similarity together, re-ranks them with a reranker, has an LLM judge relevance first, and then writes a report with citations.

Limitations and open issues

First, finding the right documents does not guarantee the model reads them well. The 'Lost in the Middle' study (2023) showed that performance is best when relevant information is at the beginning or end of the input and drops sharply when it is in the middle of a long input. This held even for models advertising long context windows. Simply attaching more documents does not solve the problem.

Second, results depend heavily on retrieval quality and store quality. As the Self-RAG paper noted, mixing in irrelevant documents can make answers worse, and as RAGStress showed, if the store's content is wrong, RAG uses that wrong content as evidence. The idea that 'it uses external documents, so it is trustworthy' holds only when the documents are trustworthy.

Third, making it smarter costs more. The Self-RAG paper observed that existing methods that retrieve on demand during generation often sacrifice runtime efficiency. The agentic RAG survey also listed evaluation, coordination between agents, memory management, efficiency and governance as open challenges. The more steps there are, the slower it gets and the harder it is to trace where things went wrong.

Further reading

If you are new to the topic, start with Lewis et al.'s 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' paper. The framework of parametric and non-parametric memory, and the difference between RAG-Sequence and RAG-Token, become the basic vocabulary for understanding later research. Reading the earlier REALM paper alongside it lets you compare a different starting point: 'attaching retrieval from the pre-training stage.'

For an overall map, the survey by Gao et al. paper is a good choice. Knowing the Naive, Advanced and Modular RAG distinction helps you place the many techniques that appear in practical material. After that, Self-RAG paper for how a model decides 'when to retrieve,' followed by the agentic RAG survey paper for agent-based architectures, makes a natural progression.

If you plan to build a real system, 'Lost in the Middle' paper and RAGStress paper are well worth reading. One gives you a sense of what order to put documents in; the other, how to test a system when the store is contaminated.