Can you trust the decision? Confidence labels, evidence, audit trails and evaluation

Why 'Confidence: Medium' deserves scrutiny

This is Part 3 of the series 'AI That Makes Decisions: The Reality and Myths of Decision RAG.' Part 2 covered a division of labor in which the LLM handles reading unstructured documents and explaining, while decision tables and policy code handle applying rules, and it ended with the point that each decision should record its rule version. This part examines whether items attached to the decision result screen, such as 'Confidence: Medium' or 'Evidence: Article 5 of the Terms,' can actually be trusted.

In decision work, confidence and evidence are not decoration. These values determine which way the workflow goes: auto-confirm when confidence is high, hand off to a person when it is low. Evidence and records are the only material that can answer 'why was this decided this way?' when a customer files an objection or an internal audit comes in. If the confidence values are wrong, the workflow goes the wrong way; if the records are empty, the duty to explain cannot be met.

The OECD AI Principles source name the values of trustworthy AI as transparency and explainability; robustness, security and safety; accountability; and human rights and democratic values, including fairness and privacy. This article turns these principles into four concrete mechanisms for a decision system: confidence calibrated through measurement, criteria for handing off to people, audit records and evaluation sets.

Why does verbalized confidence cluster at the high end?

First, some terms. A survey on confidence estimation and calibration for LLMs paper divides confidence into two kinds. Relative confidence is the ability to rank correct answers above incorrect ones. Absolute confidence means the number itself matches actual accuracy. If the cases where a model says 70% are in fact right about 70% of the time, the model is said to be 'calibrated.' The standard metric for this gap is ECE (expected calibration error). It groups confidence values into bins and takes a weighted average of how far each bin's average confidence is from its actual accuracy. The same survey also notes that ECE is sensitive to settings such as bin width, and introduces reliability diagrams, which show this as a graph.

So what happens if you ask an LLM to 'state your confidence from 0 to 100'? A study on eliciting confidence paper tested five LLMs, including GPT-4 and LLaMA 2 Chat, on five types of tasks. The confidence the models stated in words was generally overconfident, with values mostly clustered in the 80–100% range and often given in multiples of 5, as when people state their confidence. It improved as models grew larger but remained far from ideal. Prompts modeled on how people talk substantially reduced calibration error (ECE). However, failure prediction, identifying in advance which answers will be wrong, remained difficult, and every method struggled on tasks requiring specialist knowledge. A study evaluating overconfidence paper also reported that GPT-4, GPT-3.5, LLaMA2, PaLM 2 and vision-language models were mostly overconfident, with large calibration errors. This study proposed NCE (Net Calibration Error) as a metric that shows whether the error leans toward overconfidence or underconfidence, and found that calibration was poor even when models were asked to give numerical ranges such as a mean and standard deviation or a 95% confidence interval.

These results connect directly to decision system design. According to the Claude documentation source, structured outputs use constrained decoding to guarantee that responses follow a schema. Field types and required fields are respected, and the classification example in the documentation includes a confidence: float field. But this guarantee is about format. It guarantees that confidence comes out as a real number, but not that 0.8 means 'eight out of ten cases like this are correct.' The same goes for tiered labels such as 'high, medium, low.' Until you measure what accuracy 'medium' actually corresponds to, it is just a label.

The survey paper also sets out why confidence estimation is especially hard for LLMs. The space of possible outputs is huge; different wordings can mean the same thing, so outputs must be compared at the level of meaning; and results swing widely with prompt changes such as the examples or their order. There are three main ways to obtain confidence: asking the model in words, having it answer the same question several times and checking how much the answers agree, and looking at internal model information such as token probabilities.

Turning uncertainty into an action signal

The study on eliciting confidence paper split confidence estimation into three components: a prompting strategy that gets the model to state its confidence, a sampling strategy that collects multiple answers, and an aggregation strategy that computes how much those answers agree. Methods that use the consistency of multiple answers particularly improved failure prediction. White-box methods that use internal information did better, but not by much, with AUROC for failure prediction at around 0.522 versus 0.605. The researchers noted, however, that no method always beat the others. In practice, this means organizations that can only use closed APIs can still gain quite a lot from consistency-based methods. The trade-off is a higher call cost per decision.

A survey on uncertainty as an active signal paper describes a shift: uncertainty is moving from a 'diagnostic metric reviewed after the fact' to a 'control signal that changes model behavior in real time.' In reasoning, it is used to decide how much more compute to spend and whether to self-correct; in agents, to decide whether to call tools or seek more information; and in reinforcement learning, to reduce reward hacking. The survey explains these trends within theoretical frameworks such as Bayesian methods and Conformal Prediction.

In decision work, the role of confidence is less about displaying a number on the result screen and more about routing. Borrowing a distinction introduced in the survey paper, uncertainty can arise because the data itself is ambiguous (aleatoric) or because the model lacks knowledge (epistemic). The latter can be reduced by gathering more information. So a natural design sends 'uncertainty that can be reduced by looking up additional documents or re-querying the rule engine' toward tool calls, and 'cases where the input itself is ambiguous or the stakes are high' toward human review. The thresholds for each branch are not set from the numbers the model states. They are set by measuring actual accuracy by confidence band on an evaluation set.

Agents whose decisions span multiple steps are harder still. A study on uncertainty quantification for agents paper points out that most uncertainty research still stays in single-turn question-and-answer settings. It names four challenges specific to agents: which estimation method to choose, how to handle uncertainty across sources of different kinds (tool results, user responses, etc.), how to model how uncertainty changes during interaction, and the lack of fine-grained benchmarks. This means you should not assume that measuring confidence separately at each step tells you the confidence of the overall decision.

Learning when to hand off to people: learning to defer

The rule 'hand off to a person when unsure' looks simple, but it leaves something out: whether the person receiving the case actually does better on it. The learning-to-defer study paper turned this question into a learning problem. It jointly trains a classifier that predicts directly and a rejector that decides whether to pass the case to a downstream expert. Training requires only samples of decisions experts have made in the past. The researchers recast the problem as cost-sensitive learning and proposed a consistent surrogate loss function that generalizes cross-entropy loss. They also showed its effectiveness on several experimental tasks.

As an analogy, think of a junior reviewer and their team lead. A good junior reviewer does not escalate only 'what I find confusing.' They escalate 'the types of cases the team lead clearly handles better than I do.' If both find a case confusing, escalating it does not improve accuracy and only uses up the team lead's time. Conversely, if there is a type of case the junior reviewer is confident about but often gets wrong, those cases should be escalated regardless of the junior reviewer's confidence. Learning to defer says that whether to hand off should be decided not by the model's uncertainty alone but by weighing the accuracy of the model and of the person, together with the cost of errors.

Even without training a separate rejector, this idea can be applied in practice as is. First, for every case handed to a person, record the final decision the person made. As these accumulate, you can compare model and human accuracy by case type. Second, do not base the deferral criteria on a single confidence threshold. Set them by weighing case type, amount or risk tier, and how much the review staff can handle. Third, when changing the deferral criteria, version them and run regression tests, just like the rule changes described in Part 2. The problems that arise when people make judgments based on AI suggestions, and how to set the level of automation, are covered in Part 4.

Evidence and audit trails: what to record for each decision

Decision records serve two purposes. One is to explain later 'why this decision came out.' The other is to rerun the decision under the same conditions and confirm that the same result comes out. To meet both, you need to record not just the result but the inputs and versions that produced it. The items to record for each decision are as follows.

The input snapshot is the documents and input values exactly as they were at decision time. Because originals may be edited later, keep a copy or a hash. Extracted facts are the structured values the LLM read from the documents. The applied rule version is the version of the decision tables and policy code covered in Part 2. Reference documents are the identifiers of the document chunks retrieved and used as evidence, along with the versions of those documents. The model and prompt versions include the model identifier, the versions of the system prompt and examples, and the sampling settings. The confidence record keeps not just the value but how it was computed (self-reported, or consistency across how many samples) and the version of the calibration baseline used at the time. The route is where the case went: auto-confirm, additional lookup or human review. The human intervention record notes whether the case was reviewed, who reviewed it, and what was changed and why. Finally, record the final result communicated to the customer.

Do not stop at 'the evidence is there'; verify it. Add a step that automatically checks whether the clauses or document chunks cited in the decision explanation are actually in the list of reference documents, and whether those chunks actually contain what is claimed. Like the confidence field, the evidence field only guarantees format; whether the content is correct has to be checked separately.

Reproducibility checks use these records. Rerun the stored input snapshot with the same rule, model and prompt versions and see whether the decision is the same. The part handled by the rule engine should be identical every time. The extraction and explanation parts handled by the LLM can produce different outputs for the same input, so check whether the rate at which the final decision changes is within an acceptable range. As the survey paper points out, LLMs are sensitive to prompt changes such as the examples and their order. So even a small prompt change should be recorded as a new version, and the reproducibility check should be run again.

Evaluation: how to measure decisions

The starting point is an evaluation set (Evals) with ground-truth decisions. Collect past cases that experts have confirmed, but do not include only easy ones. In the confidence-elicitation study paper, every method struggled on tasks that required expert knowledge, so deliberately include cases that are hard to judge in practice and borderline cases. To evaluate the deferral criteria, also record the decision a person made on each of those cases.

This set measures more than accuracy alone. The first thing is calibration. Measure actual accuracy for each confidence bin, report it with ECE and a reliability diagram, and use a directional metric such as NCE to check whether the model leans toward overconfidence paper. The second is failure prediction. Look at how well confidence separates correct decisions from incorrect ones, using a metric such as AUROC. In the confidence-elicitation study paper, calibration error went down while failure prediction stayed the same, so these two can move independently. The third is consistency. Check whether a decision holds when the same case is fed in with only the wording changed or only the document order changed. The fourth is differences between groups. For each meaningful segment, such as customer group, region or intake channel, look separately at accuracy, auto-confirmation rate and human deferral rate, and find differences hidden by the overall average. The fifth is drift over time. When the format of input documents, the rules or the model version change, calibration criteria set earlier can break. Periodically add cases confirmed by people during operation to the evaluation set and recheck calibration.

At the organizational level, the NIST AI RMF can serve as a reference. According to the NIST page source, the AI RMF is a voluntary framework that helps incorporate trustworthiness considerations into the design, development, use and evaluation of AI products, services and systems. It was released on January 26, 2023, together with the Playbook, a companion implementation guide. On July 26, 2024, the Generative AI Profile (NIST-AI-600-1) was released. On April 7, 2026, a concept note for a critical infrastructure profile was published, and AI RMF 1.0 is being revised in line with the White House AI Action Plan. The page excerpts this article relied on do not include the detailed items of the Measure and Manage functions. So this article does not assert how the evaluation items above map to specific functions. After setting up measurements of calibration, consistency, group differences and drift, along with management mechanisms such as deferral criteria and recalibration, it is recommended to read the Playbook directly and map it to your company's risk management items.

Common mistakes checklist

The most common mistake is treating a JSON confidence field or a 'high/medium/low' label as a probability without measuring it. The mistake that follows from this is setting thresholds by gut feeling. A rule like 'auto-confirm if 0.8 or above' only means something after you have checked the actual accuracy of that bin on the evaluation set. Recalling that verbalized confidence in the study clustered at 80–100%, a threshold set by gut feeling could let almost every case through.

The second group is cases where people mistakenly believe the problem is solved. Even if prompting makes the model more modest and reduces calibration error, failure prediction may stay the same. It is also common for deferral criteria to look only at the model's confidence, without considering whether people are better at that type of case or how much volume reviewers can handle. Carrying over evaluation results from single-turn question-and-answer settings directly to the decisions of an agent that operates over multiple steps is the same kind of mistake paper.

The third group is gaps in record-keeping and operations. Cited content is not verified just because there is an evidence field. Model, prompt and rule versions are missing from the records, so results cannot be reproduced. The evaluation set is built once and never updated, so drift goes unnoticed. Only overall accuracy is checked, not differences between groups. Check the list below before deployment and during regular reviews.

Further reading, and on to Part 4

To get the concepts down first, read Chapter 2 (basic concepts and metrics) of the survey on LLM confidence estimation and calibration paper. It covers relative and absolute confidence, ECE and the two types of uncertainty. Next, the confidence-elicitation study paper shows through experiments what can be done in environments that use only closed APIs. The overconfidence evaluation study paper is short, so it is enough to check the NCE metric and the calibration problem across numeric ranges.

For how to build uncertainty into system design, continue with the design patterns section of the survey on uncertainty as an active signal paper, and for the difficulties that arise when extending to agents, the study on agent uncertainty quantification paper. The learning-to-defer study paper is math-heavy. Reading only the problem setup and the classifier–rejector structure is enough to understand the deferral criteria in this article. On the policy side, check AI RMF 1.0 and the Playbook on the NIST AI RMF page source, and the values-based principles in the OECD AI Principles source. What structured outputs do and do not guarantee can be checked directly in the Claude documentation source.

This part focused on turning confidence and evidence into measurable values. But even after measurement, questions remain: 'Which decisions should be left to people?', 'How much do people follow AI suggestions?', and 'How do regulations require a human role?' Part 4, 'Where do people stand: automation levels, regulation and cases', addresses these questions.