Where do humans stand? Hype and reality seen through decision automation levels, regulation and cases

Closing the series: why the last question is 'the place of humans'

This article is the fourth and final part of the series 'AI that decides: hype and reality of Decision RAG'. The earlier parts can be summed up in one paragraph. Parts 1–2 looked at the structure and uses of Decision RAG, which finds rules and documents, attaches them as evidence and makes a decision. Part 3 covered how the confidence that LLMs state in words tends to skew high, so the confidence field in JSON does not directly become a probability; how uncertainty should be used as a signal to route cases to 'auto-confirm, additional lookup or human review'; and how each decision should leave an audit record of its inputs, rule version, evidence and human intervention.

One question remains. How far should such decisions be left to machines, and where should humans stand? This question is closer to an issue of organization and regulation than of technology. The same model can be a tool that cuts customer wait times when used to approve small insurance claims, yet can directly harm patients when used as grounds to cut off a period of care.

Focusing on finance and healthcare (including insurance), this article first divides automation into three levels, then sets out what EU and US regulations require, and what worked and what went wrong in real cases. The regulatory descriptions summarize published provisions and press releases and are not legal advice. Whether they apply in practice must go through each organization's legal review.

Three levels of automation: automated decision, assistance and escalation

When designing decision work, the question is not 'whether to use it' but 'which level each case goes to'. This article uses three levels. The first is automated decision. AI handles cases end to end when they are low-risk, small in amount, governed by clear rules and easy to reverse if wrong. People sample and inspect them afterward.

The second is assistance. AI produces a draft decision and a summary of the evidence, and a person makes the decision. The key here is not to show people only the conclusion. The evidence documents, the rules applied, the evidence pointing the other way and the confidence signals covered in Part 3 must be shown together so that people can actually 'judge'. The third is escalation. Decisions that are high-risk, involve ambiguous facts, are classified as high-risk under regulation, or go against the applicant are sent up to experienced staff, who are also given ample review time.

The practical principle this article proposes is short. Start automation on the 'approve' side and leave 'deny' to people. Wrongly approved small claims can be caught by after-the-fact inspection, but for someone wrongly denied, the harm stays unless they file an appeal. Also, do not let the model decide on its own which level a case goes to; decide using external rules such as amount, product and decision direction together with measured confidence.

What EU regulation says: the AI Act and GDPR Article 22

Annex III of the EU AI Act source contains the list of high-risk AI systems. The item that directly touches finance and insurance is point 5, 'access to essential private and public services'. It covers systems that evaluate the creditworthiness of natural persons or establish their credit score (excluding those for detecting financial fraud); systems used for risk assessment and pricing of natural persons in life and health insurance; systems used by public authorities to evaluate eligibility for public benefits, including healthcare services, or to grant, reduce or revoke such benefits; and emergency patient triage systems. Recruitment and HR decisions (point 4) and educational assessment (point 3) are on the same list.

Article 14, 'Human oversight' source, requires that high-risk systems be designed and developed so that people can oversee them 'effectively' while they are in use. The purpose of oversight is to prevent or minimize risks to health, safety and fundamental rights, and oversight measures must be commensurate with the risks, level of autonomy and context of use. There are two kinds of measures: those the provider builds into the system in advance, and those the provider identifies and the deployer (the organization actually using the system) implements. According to an article commentary site source, this provision applies to Annex III high-risk systems from December 2, 2027.

Article 86 source is the right to an explanation of individual decisions. A person who considers that a decision taken by a deployer on the basis of output from an Annex III high-risk system (excluding point 2, critical infrastructure) produces legal effects or similarly significantly harms them can request a clear and meaningful explanation of the role the AI played in the decision-making process and the main elements of the decision. The same site states that this provision applies from August 2, 2026. If other EU law already grants the same right, that law takes precedence.

GDPR Article 22 source is an earlier rule. It establishes the right not to be subject to a decision based 'solely on automated processing', including profiling, that produces legal effects or similarly significant effects. There are exceptions where the decision is necessary for performing a contract, authorized by law or based on explicit consent, but even under the contract and consent exceptions, at minimum the rights to obtain human intervention, to express one's point of view and to contest the decision must be guaranteed. In practice, the sticking point is the word 'solely'. Whether a procedure in which a person checks only as a formality can be recognized as not being an automated decision is a matter of interpretation, so legal review should be obtained at the design stage.

US bank supervision: the space the OCC guidance left empty

In an April 17, 2026 press release source, the US Office of the Comptroller of the Currency (OCC) announced that it had issued new model risk management guidance together with the Board of Governors of the Federal Reserve System (the Fed) and the Federal Deposit Insurance Corporation (FDIC). It rescinded earlier documents, including the 2011 supervisory guidance on model risk management (Bulletin 2011-12) and the 1997 guidance on examining credit scoring models, and stated that model risk management should be risk-based, matched to a bank's size, complexity and extent of model use. The guidance states explicitly that it is not an enforceable standard and that not following it will not result in supervisory criticism. It also said it is most relevant to banks with more than $30 billion in total assets.

The sentence that bears directly on this article's topic is this one. The OCC stated that generative AI and agentic AI models are novel and rapidly evolving and therefore fall outside the scope of this guidance. Instead, the three agencies said they plan to issue a request for information (RFI) in the near future on banks' use of AI, especially generative and agentic AI.

In practice, this should not be read as 'no regulation, so anything goes'. LLM-based decision systems are in a state where they neither fit neatly into the existing model risk management framework nor fall outside it. To prepare for future supervisory direction, it is safer to start now keeping decision records, validation and human intervention at a level equal to or higher than for existing models.

The 'reality' seen in cases: small-value, high-volume, clearly rule-based work

Allianz Project Nemo (Australia, claims processing). According to a report in the industry outlet Coverager source, Allianz launched its first agentic AI system in Australia in July 2025 to handle food spoilage claims (food spoiled by power outages and similar events) that surge during natural disasters such as storms. Seven specialized agents covering coverage checks, weather verification, fraud screening and payout calculation work together, processing claims under AUD 500 within a day, usually within a few hours. Importantly, the final payout decision is still made by a human claims adjuster. The figure of an 80% reduction in processing and settlement time is a company statement made by Allianz Services' chief innovation officer and has not been independently verified.

Lemonade (US, automated claims processing). A Lemonade company blog post source announced that its claims bot 'A.I. Jim' reviewed, approved and issued payment instructions for a claim on a stolen coat in 3 seconds. Within those 3 seconds, it reportedly cross-checked the policy and ran 18 anti-fraud algorithms. In the same post, the company also wrote that A.I. Jim frequently hands claims off to human staff, so not every claim is processed instantly. In other words, the structure combines automatic decisions with escalation.

What the two cases have in common is that the amounts are small, the volume is high, and the decision rules (coverage scope, weather records, deductibles) are clear. They are also alike in that automation is mainly aimed at fast 'payouts'. This ties in with the 'automate approvals first' principle discussed in Section 2. However, both sources are company announcements or reports relaying company statements, so it should also be kept in mind that counter-indicators such as error rates or later disputes cannot be confirmed from public sources.

Warning cases: problems caused by 'rubber stamp' oversight

UnitedHealth lawsuit (US, health insurance). According to STAT reporting source, a class action was filed in November 2023 against UnitedHealth Group and its subsidiary NaviHealth. The plaintiffs allege that an algorithm predicting patients' length of care (nH Predict) was used to systematically deny payment for rehabilitation care for seriously ill patients. The complaint claimed the algorithm had a 90% error rate, based on the share of denials overturned on appeal. These are allegations raised in the lawsuit, not facts established by a court. The company countered that the tool is not used to make coverage determinations and that the lawsuit has no merit, and stated that final decisions are made by physician reviewers. However, earlier STAT investigative reporting stated that the company set a target for managers to keep rehabilitation stays within 1% of the algorithm's predictions, and that the denial recommendations physician reviewers received were submitted by staff subject to this target.

Cigna reporting (US, health insurance). A 2023 investigation by ProPublica and The Capitol Forum source examined Cigna's review process called PXDX. When the algorithm flagged claims where the diagnosis did not match the tests and procedures the company accepts, company physicians reportedly signed off on denials in batches without opening patient records. Citing internal documents, the report said more than 300,000 claims were denied over two months, at an average of 1.2 seconds per claim. Cigna countered that the reporting was biased and incomplete, and said the system was built to speed up payment of routine screening claims and only decides whether to pay after care is provided, without blocking the care itself.

What the two cases have in common is that a structure in which 'a human makes the final decision' existed on paper. The questions raised were whether that person actually had the time, information and authority to make a judgment, and whether they faced no penalty for reaching a conclusion different from the algorithm. This is the 'rubber stamp' problem. It is also why Article 14 of the EU AI Act source requires not just 'oversight' but 'effective' oversight. Both cases are also alike in that the decisions went in the direction of 'denial', and correcting the harm depended on the affected person filing an appeal.

Hype and reality: what is a real effect and what is an illusion

There are three kinds of hype. First, decision automation that is new in name only. Calling it an 'AI agent' or 'Decision RAG' adds no new value if it ultimately repeats, at higher cost, what an existing rules engine already did, or if it is a tool that churns out denials in bulk like the warning cases above. Second, the illusion created by a confidence field. As covered in Part 3, when a screen shows 'confidence 0.92', people trust the number and look less closely. Uncalibrated numbers weaken human oversight. Third, oversight that consists only of the phrase 'a human makes the final decision'. Without review time, access to information, and a path for handling disagreement, that phrase is merely a device for shifting responsibility onto a person.

There are also three real benefits. First, shorter processing times for small-value, high-volume work with clear rules. The Allianz and Lemonade cases sourcesource fall into this area. Second, drafting and organizing evidence. If a human decides while AI gathers and organizes relevant rules, documents and past cases, human review can actually become deeper. Third, decisions that leave a record. If each decision records its inputs, evidence, rule version and any human changes, it becomes easier to explain the 'main elements of the decision' referred to in Article 86 of the EU AI Act source, and appeals can also be handled.

Common mistakes are also worth listing: setting human reviewers' processing targets (volume, time) in terms of agreement with the algorithm's output; not measuring at all how often humans judge differently from the AI; not feeding the rate of decisions overturned on appeal back into model evaluation; and defining scope solely by the label 'assistive tool' without checking whether the use falls on the high-risk list source.

Adoption steps, checklist, and further study

The order for starting small is as follows. 1) Build a list of tasks and classify them by amount, volume, clarity of rules, ease of reversal, and regulatory applicability (Annex III, GDPR Article 22). 2) At first, route every case through the 'assist' tier, recording the AI draft and the human decision side by side. 3) Collect cases where the human and AI judged differently, and cases overturned on appeal, and examine the causes. 4) Move small-value approvals, where agreement and calibration have been sufficiently confirmed, to 'automatic decision' first, and add post-hoc sample audits. 5) Keep denials, high-value and ambiguous cases in 'escalation', and review the tier criteria regularly.

The items to check are these: Can human reviewers see the supporting documents and counter-evidence? Is the time needed for review reflected in work targets? Is there no penalty for judging differently from the AI? Are the rate of human overrides and the outcomes of appeals being measured? Can the AI's role and the main elements of the decision be explained to the affected person? Are decision logs kept in a reproducible form? For US banks, also track the fact that generative and agentic AI currently fall outside the scope of OCC guidance source, along with the announced request for comment.

For further study, it is best to start with the legal text. Use Annex III of the EU AI Act source to check whether your work falls on the high-risk list, then read Article 14 source, Article 86 source and GDPR Article 22 source together to see how 'oversight', 'explanation' and 'appeal' fit together. On the US side, the original OCC press release source is short. For the cases, read the STAT source and ProPublica source articles alongside the companies' rebuttals, and ask yourself whether your organization's review process could face the same criticism. This concludes the 'AI that makes decisions' series.