GraphRAG, knowledge graphs and ontologies: turning a pile of documents into a relationship map you can query

Why it was needed: the limits of chunk retrieval

RAG (retrieval-augmented generation) is a way to handle a document collection larger than what an LLM can read at once, that is, its context window. Documents are split into small pieces (chunking), each chunk is stored as an embedding (a numeric vector representing the meaning of the text), and a few chunks close in meaning to the question are retrieved and passed to the LLM along with it. The GraphRAG paper paper calls this standard approach 'vector RAG' and notes that it works well for questions whose answers are concentrated in a small number of records.

The problem is questions like 'What are the main themes of this whole dataset?' Such a question is not a retrieval problem of finding specific chunks but a task of summarizing the entire corpus in light of the question (query-focused summarization, QFS). Retrieving only a few chunks similar to the question cannot give the full picture. Conversely, existing summarization techniques did not scale to the volume of text that RAG systems index. GraphRAG targeted the gap between the two.

Another case is multi-hop questions that require connecting facts scattered across several documents. GRAG paper points out that on interconnected documents such as citation networks, social media and knowledge graphs, existing RAG is weak because it looks only at individual documents. The roadmap by Pan et al. paper identifies a more fundamental reason: LLMs hold knowledge implicitly in their parameters, so they produce hallucinations that contradict the facts, and it is hard for people to check the grounds for what they say.

Core idea: redrawing documents as a relationship map

An analogy is what a detective does when handed a thick case file. Instead of rereading the file from the beginning, the detective pins people, places and events on a wall and connects their relationships with string. Then, for each cluster where the strings are densely tangled, they write a note on 'what this cluster is about'. When someone asks 'What is the big picture of this case?', the detective skims those notes to answer. In GraphRAG, the relationship map is the knowledge graph, the clusters are communities, and the notes are community summaries.

A knowledge graph is a data structure that represents facts as points (nodes) and lines (edges). The roadmap paper paper explains that knowledge graphs store facts as triples in the form (head entity, relation, tail entity). For example: (Newton, formulated, theory of gravity). Because the knowledge is explicit, accuracy and interpretability are high, but knowledge graphs are hard to build and always incomplete.

An ontology is the blueprint for that graph. It defines what kinds of entities exist (people, organizations, drugs) and which relations are allowed between them (affiliation, side effect). The introduction by Hogan et al. paper considers the roles of schema, entity identity and context in knowledge graphs important enough to treat as separate topics. GraphRAG's early approach is closer to having the LLM freely extract entities and relations from the source text, and approaches that constrain extraction with an ontology are reappearing in recent research.

How it works: two stages, indexing and querying

GraphRAG paper splits the work into 'indexing', done before a question arrives, and 'answering', done after it arrives. In the first indexing step, an LLM extracts key entities and the relationships between them from the source documents to build an entity knowledge graph. Next, graph community detection (an algorithm that finds groups of densely connected nodes) divides the graph into hierarchical communities, with smaller topics nested under larger ones.

The LLM then writes a summary for each community. Summaries are built from the bottom up, and each higher-level community summary is written by combining the summaries below it. Together, these summaries act as a 'table of contents and commentary' for the whole corpus. When a question arrives, it is answered with a map-reduce approach. In the map step, partial answers are generated independently and in parallel, each based on a community summary. In the reduce step, the partial answers are combined into the final answer.

Not all graph-based RAG works this way. For multi-hop questions, GRAG paper finds 'textual subgraphs' that match the question. It uses a divide-and-conquer strategy to find the optimal subgraph in linear time, then gives it to the LLM in two views: the subgraph written out as text, and the graph structure itself. In short, GraphRAG is designed for 'questions that summarize the whole', and GRAG for 'questions that follow connections'.

How the field developed: from knowledge graphs to graph RAG

The starting point is work that organizes knowledge graphs themselves. Hogan et al. 2020 paper gave a broad overview of the data models and query languages used for knowledge graphs; the roles of schema, identity and context; deductive and inductive techniques for representing and extracting knowledge; and methods for creating, enriching, assessing the quality of, refining and publishing graphs. Later research combining knowledge graphs with LLMs draws its shared vocabulary from here.

After LLMs arrived, Pan et al. 2023 paper laid out a roadmap for combining the two in three directions: using knowledge graphs to enhance LLMs (KG-enhanced LLM), using LLMs to help build, complete and answer questions over knowledge graphs (LLM-augmented KG), and having the two strengthen each other as equals (Synergized LLM + KG). The same year, LLMs4OL paper tested whether LLMs can do 'ontology learning', that is, automatically extracting a system of concepts from text. It evaluated nine LLM families with zero-shot prompting (giving only instructions, without examples) on three tasks: term typing, taxonomy discovery and non-taxonomic relation extraction. It covered knowledge from WordNet (lexical), GeoNames (geographic) and UMLS (medical).

In 2024, attempts to build structure on the retrieval side continued. RAPTOR paper repeatedly embeds text chunks, clusters similar ones and summarizes them, building a summary tree from the bottom up, and retrieves at different levels of abstraction. Combined with GPT-4, it reported improving the best performance on the QuALITY benchmark by 20% in absolute accuracy. The GraphRAG paper paper describes itself as similar to hierarchical summary indexes such as RAPTOR, and names as its difference that it builds a graph index from the source text and uses community detection to divide it by topic. GRAG paper then proposed retrieving subgraphs from connected documents.

The evaluation approach was also new. Because questions about overall themes have no fixed correct answer, the GraphRAG team used an LLM-as-a-judge approach: one LLM generated questions tailored to the corpus, and another LLM compared the answers from two RAG systems against predefined criteria. They report that on datasets of about 1 million tokens, GraphRAG clearly outperformed vector RAG in the comprehensiveness and diversity of its answers.

Where things are heading

First, there is a move away from 'the same search method for every question'. MOSAIC paper identifies as a problem that most GraphRAG systems use a similar traversal procedure regardless of the question. A simple factual question needs only nearby neighbors, a comparison question needs to cover several targets evenly, and an indirect question needs deeper paths. So, without additional training, an LLM analyzer sets the policies for choosing starting points, traversing the graph, stopping and selecting evidence for each question. In comparisons using the same graph and generator, none of the fixed policies (narrow, medium or broad) was always best. MOSAIC reports scoring 9.96 points higher than the strongest fixed policy while evaluating 81.9% fewer paths than the 'broad' policy.

Second, there is a move to bring ontologies and constraints back into the graph. NS-ST-GraphRAG paper handles texts such as long literary works, where evidence is scattered across many chapters and relationships change over story time. It adds ontology-aligned extraction, rule-based constraint checking, temporal coordinates and spatial attributes of scenes. Instead of a single fixed graph, it selects and searches the graph state that matches the question's temporal and spatial scope. If LLMs4OL, covered above, asked 'can LLMs build ontologies?', this work asks 'can ontologies constrain LLM extraction?'

Third, as GraphRAG has become part of real systems, security has become a research topic. A study paper showed that auxiliary structures built in advance during indexing, such as summaries and scores, can become attack targets. This is covered in detail under 'Limitations and open issues'.

Limitations and open issues

The first issue is the trade-off between cost and quality. Because GraphRAG builds its graph and community summaries with an LLM, a substantial amount of LLM processing must be done before any question arrives, and graph quality depends directly on extraction quality. As the roadmap paper paper points out, knowledge graphs are inherently hard to build, incomplete and constantly changing. Graphs built by automatic extraction are not free of these weaknesses either.

The second is security. The HDI study paper notes that earlier attacks targeted only individual elements such as nodes, edges and triples, and defines auxiliary structures, such as semantic summaries, hierarchical edges and precomputed scores, as a new attack surface. In tests on Microsoft GraphRAG and HippoRAG2 with HotpotQA and 2WikiMultiHopQA, changing just 0.016% of the auxiliary structures produced attack success rates of 88–94%, and a single modification affected up to 6.00 questions. Because the manipulated summaries read as naturally as system-generated text, they evaded perplexity-based and paraphrasing-based defenses more than 99% of the time. The study argues that a design that trusts prebuilt structures at run time without verification is itself a blind spot.

The third is evaluation. Questions about overall themes have no correct answer, so the original GraphRAG paper also had an LLM compare and score answers. This approach is practical, but the results depend on the judging LLM's criteria. Some recent studies report results cautiously. NS-ST-GraphRAG paper states that in a 120-question evaluation, answer recall was 0.733 versus 0.675 for the baseline, a favorable direction but not statistically significant (p = 0.092), and notes that one of its pre-stated hypotheses was not supported. It is still too early to conclude that adding a graph always helps.

Further study

If you need conceptual grounding, start with Hogan et al.'s 'Knowledge Graphs' paper to learn basic vocabulary such as data models, schemas and quality assessment. Then read Pan et al.'s roadmap paper to learn how to classify research linking LLMs and knowledge graphs into one of its three directions.

Next comes the original GraphRAG paper paper. It helps to look at the indexing → community summary → map-reduce answering flow together with its LLM-as-a-judge evaluation design. For comparison, reading RAPTOR's paper summary tree alongside GRAG's paper subgraph retrieval shows how the same idea, 'build structure, then retrieve', branches out depending on the type of question.

If you are interested in ontologies, read LLMs4OL paper followed by NS-ST-GraphRAG paper. If you build real systems, we recommend reading MOSAIC's paper per-question traversal policies together with HDI's paper analysis of attacks on auxiliary structures. Together they show both a design that improves performance and the risks that such designs open up.