What is Decision RAG: from AI that explains to AI that decides

Why it emerged: RAG that stops at explanation

This article is Part 1 of the series 'AI That Decides: The Hype and Reality of Decision RAG'. As the introductory installment, it starts by laying out where the term came from and what it claims.

RAG (retrieval-augmented generation) first finds documents relevant to a question and then feeds their content to the model so it produces a grounded answer. The RAG survey paper paper explains that RAG emerged to address LLM weaknesses such as hallucination, outdated knowledge and hard-to-trace reasoning by drawing on knowledge from external databases. But much enterprise work does not end at 'what does this policy say'. It requires deciding whether to approve a purchase request, which team to assign a ticket to, or whether to hand it to a person (escalation).

The name 'Decision RAG' comes from a LinkedIn column by Martin Connell source. The column says most RAG is designed to 'inform', not to 'decide'. Even advanced implementations, it argues, ultimately stop at interpretation and fail to deliver explicit decisions, consistent rule application, structured output, operation within defined control boundaries and repeatable results. The column, however, is an opinion piece written without supporting papers or real cases. This series treats Decision RAG as 'one practitioner's proposal' and does not use it as an established industry term.

The same concern appears elsewhere. A post by software developer Leanware source notes that RAG generally produces grounded answers but gets stuck on support ticket routing, loan approval and yes/no compliance decisions. The same question can yield different results on each run, there is no guarantee that business rules are followed, decision records (logs) are not structured, and there is no path to execute workflows. That post calls the same structure a 'retrieval-augmented decision engine'. The fact that no single name has emerged is itself a sign that the concept is still taking shape.

Where the decision-oriented type sits in the RAG lineage

What the decision-oriented type newly demands is easiest to see when it is set alongside earlier branches of RAG. The RAG survey paper paper divides RAG into Naive (basic) RAG, Advanced RAG and Modular RAG, and organizes techniques along three axes: retrieval, generation and augmentation. The agentic RAG survey paper holds that conventional RAG is tied to fixed workflows and cannot do multi-step reasoning, and it describes agentic RAG, which dynamically changes retrieval strategies through patterns such as reflection, planning, tool use and multi-agent collaboration.

Comparing the four branches by input, output, reasoning method and explanation responsibility gives the following. Basic RAG takes a single question, finds relevant document chunks and writes an answer for a person to read. Reasoning happens implicitly inside the model, and showing sources is nice but not required. Advanced RAG has the same input and output shape, and improves the quality of retrieved evidence with rerankers (re-ranking models) or hybrid search. Agentic RAG takes a single goal and repeats retrieval and tool calls many times; the longer that process gets, the harder it becomes to trace why it did what it did. The agentic RAG survey paper also lists evaluation, orchestration and governance as open challenges.

The decision-oriented type described in the column source takes as input a single case (an application, ticket or transaction) together with its attached documents, structured records and policies. The output is a structured record containing the decision and its grounds. Reasoning follows explicit criteria ('evaluate against these conditions'), and explanation is mandatory, not optional. The column summarizes this progression as 'search returns documents, RAG returns answers, Decision RAG returns outcomes.'

There is a caveat about this lineage. Basic, advanced, modular and agentic RAG are classifications compiled by survey papers, while the decision-oriented type is a slot added by a single column. They should not be treated as classifications of equal weight.

The five elements and the corresponding research and tools

The column source describes the decision-oriented type in terms of five elements. Condensed, and paired with the research and tools that actually correspond to each element, they are as follows.

First, it combines context from multiple sources. This means cross-checking not only document chunks but also structured records, policies and operational data. The Leanware post source describes this stage as the retrieval layer: unstructured knowledge comes from a vector DB, structured records from SQL databases, real-time state from APIs and policies from document stores. Palantir's AIP Logic documentation source also gives an example of an LLM-based function that takes ontology objects as input and cross-checks them against customer emails.

Second, it makes decision criteria explicit. Instead of asking 'what does this information suggest', it instructs 'determine the matching outcome against these conditions'. RAD paper comes closest here. This work points out that existing RAG lacks quantitative weights and traceable reasoning paths. It therefore automatically extracts key criteria from industrial documents to build a weighted hierarchical decision model, and writes structured reports according to that model. The analytic hierarchy process (AHP), originally a multi-criteria decision-making technique, was hard to scale because experts had to design the criteria by hand; this work hands that task to an LLM and multiple agents.

Third, it outputs results as structured output. OpenAI (w3) and Anthropic source both provide structured output features in their API documentation. The Claude documentation source explains that the feature uses constrained decoding to guarantee that responses follow a specified JSON schema. There are two forms: JSON outputs, which set the response format, and strict tool use, which validates tool inputs against a schema.

The fourth and fifth are built-in rules and controls, and mandatory explanation. Policy-as-Skill paper addresses both together. This work does not leave policies only as text inside prompts but packages them as versioned executable units (skills). A single skill defines a name, version, retrieval scope, required evidence, decision schema, conditions for handing off to human review, audit log fields, behavior when evidence is missing, a prompt template and a context contract. The researchers compared 13 methods on 600 development tasks using a fixed Gemma4 model. The configuration with audit features added (PaS+Audit) recorded exact-match accuracy of 53.8%, review-routing F1 of 0.854, citation precision of 1.000 and audit completeness of 1.000, and outperformed LLM+RAG on most governance and review metrics.

How it works: an example decision output

The flow of a decision-oriented system is easiest to follow using the layered structure laid out in the Leanware post source. The retrieval layer gathers context from multiple sources. The LLM in the reasoning layer does not make the final decision; it produces signals such as classification labels, confidence scores, extracted fields and risk assessments. The decision logic layer applies thresholds, policy constraints, regulatory rules and escalation conditions to these signals. This layer is deterministic: the same input yields the same output. Finally, the execution layer updates records or triggers workflows, and every step is kept as an audit log.

Below is an example, constructed for this article, of the decision on a single internal purchase request. It is not output from a real system.

{"case_id": "PR-2026-1107", "decision": "Route to human review", "policy": {"name": "Purchase approval policy", "version": "v3"}, "criteria": [{"rule": "Budget balance check", "result": "Met"}, {"rule": "Within approval limit", "result": "Exceeded"}, {"rule": "Registered supplier", "result": "Met"}], "evidence": [{"source": "Purchase approval policy v3, Section 4.2", "quote": "Requests above the limit require department head approval"}, {"source": "ERP budget record"}], "next_step": "Assign to department head approval queue", "needs_human_review": true}

Each field has a job. decision is the value downstream systems branch on. The policy version is kept so that you can later trace 'which policy the decision was based on at the time'. Policy-as-Skill paper records policy and model versions and evidence IDs in its audit logs for the same reason. criteria records separately whether each criterion was met, showing the 'why' line by line. evidence points to the location in the source text that was referenced, and next_step and needs_human_review are the connection points to the execution layer and the human review flow.

There is a reason the decision in this example is 'route to human review' rather than 'approve' or 'reject'. The column source also says the decision-oriented type is not about removing people but about automatically handling low-risk cases, assisting with intermediate ones and handing risky or ambiguous ones to people. Policy-as-Skill paper likewise predefines failure behavior so that cases with insufficient evidence are treated as 'unknown' or 'review'.

The first sign of hype: a new name, old work

Take the decision-oriented type's structure apart and there is less that is new than you might expect. Keeping business rules separate from code and making the same input produce the same result is what rule engines, business rule management systems (BRMS) and decision tables (decision modeling standards such as DMN, Decision Model and Notation) have long done. The Leanware post source also says the decision logic layer is implemented with rule engines, scoring models or simple conditionals. If so, what is new in the decision-oriented type is less the 'decision' itself than the fact that an LLM handles the front end: extracting decision criteria and evidence from scattered documents and feeding them into rules. That front end, criteria extraction and weighting, is exactly what RAD paper does.

The second sign is the cost of structured output. The guarantee described in the Claude documentation source is a format guarantee that the response follows the schema, not a guarantee that the decision is correct. A study on format restrictions paper reported that forcing formats such as JSON or XML significantly degrades LLM reasoning ability, and that stricter format constraints generally lead to larger performance drops on reasoning tasks. This is why the more a system depends on receiving a decision JSON, the more it needs to think separately about an order of 'reason freely first, then put it into the format'.

The third sign is that the effect of rules is uneven. In Policy-as-Skill paper, turning on deterministic controls raised overall accuracy to 61.2%, but the researchers noted that the effect depends heavily on task type. They therefore conclude that rule interventions should be applied selectively rather than everywhere. The same study warns that simply emitting lots of metadata can make composite scores look better. 'Having an explanation attached' and 'the decision being correct' are different matters.

One more point to avoid confusion. This site's 'Decision model' article covers a type of model that assigns probabilities to each option and picks one directly. The decision-oriented RAG in this article refers to a system design that combines retrieval, rules and structured output, so the two are different concepts.

Where it is heading now

Recent research shows concrete results from the idea of "moving decision logic outside the LLM." GDGraph paper, a study of decision graphs built from radiology guidelines, separates the step of organizing guidelines from the step of evaluating individual cases. First, an LLM converts clinical guidelines scattered across prose, tables and cross-references into executable decision graphs. When a case is evaluated, a fast model answers only local questions, and deterministic graph traversal handles the true/false logic. Across 347 cases covering guidelines for renal cysts, adnexal masses and thyroid nodules, accuracy rose from 57.3% to 90.8%, higher than retrieval-augmented GPT-4o (80.1%) and graph-guided GPT-4o with CoT (82.7%). The researchers also reported much lower latency and cost. However, these results come from a specific domain, medical guidelines, and should not be read directly as real-world clinical performance.

Another direction is to accumulate human decisions themselves as evidence. iRAG paper is a system whose knowledge base starts out empty; it stores decisions made by people as precedents and uses them for retrieval. It starts by quietly observing, moves to a stage where it challenges human judgments with evidence, and then shifts to automated processing that can abstain from deciding. The researchers reported that a mechanism that down-weights older precedents improves performance when knowledge changes but increases errors when knowledge is stable. In other words, it is a trade-off, not a general advantage. They noted that usability with real users has not yet been evaluated.

The two studies point to the same picture. The LLM is used to read documents, extract criteria and make local judgments, while the backbone of the decision sits in traceable structures such as graphs, rules and precedents. A study on the evolution of AI in finance paper concludes that more advanced models or autonomous agents are not better at every task, and that the approach should be chosen to fit the nature of the task. That advice applies directly to decision-style RAG as well.

When to use it and when to avoid it

Decision-style design fits work where the decision criteria exist in documents, the outcome falls into a small number of options (approve, reject, review, assign), and someone later needs to explain who decided what and why. Compliance classification, internal approvals and ticket routing are examples. Conversely, if the criteria themselves have not yet been agreed on, or if an explanation that a person reads and judges is a sufficient answer, conventional RAG will do. Adding a decision layer only adds more to manage.

There are four common mistakes. First, pasting policy text into the system prompt and assuming that creates control. Policy-as-Skill paper points out that a natural-language prompt alone does not determine which policy version to use, what to do when there is no evidence, or when to hand off to a person. Second, mistaking schema-compliant output for a correct decision paper. Third, believing that output is protected when access control applies only to input. The Palantir documentation source states explicitly that row- and column-level access control filters the data the model reads, but does not apply to the model's output or to edits the model makes. Fourth, using a decision's confidence score directly for thresholds without validating it. The Leanware article source recommends continuously monitoring confidence calibration, changes in approval rates, and decisions overturned by people.

The most important principle is that people hold final responsibility. The goal of decision-style systems is consistency and traceability, not handing responsibility for judgment over to the system. The conditions for routing to human review and the behavior when there is no evidence must be built into the design from the start.

Further study, and on to Part 2

The starting point is Martin Connell's column source. It lets you quickly grasp the backbone of the argument, but read it keeping in mind that it is an opinion piece. Next, the layer structure and the comparison table against RAG in the Leanware article source will give you a feel for the design. If you want to understand the lineage precisely, read the RAG survey paper and then the agentic RAG survey paper.

To check each element of decision-style systems against research, continue with RAD paper for criteria extraction and weighting, and Policy-as-Skill paper for control, auditing and human review. Policy-as-Skill in particular includes an analysis showing that deterministic control works differently from task to task, which leads directly into the question of the next part. Before adopting structured output, be sure to also read the study on format constraints paper. GDGraph paper and iRAG paper illustrate recent trends well.

Part 2 tackles head-on the question this article leaves open: should decision rules be left to the LLM or to a rules engine? And where should that boundary be drawn?