A Human-Centered, Risk-Constrained Evaluation Framework for Financial Document Recognition: Evidence, Critical Errors and Selective Automation

Problem

Financial document recognition is often evaluated with CER and WER. But one misspelled letter in a narrative sentence cannot be treated as the same error as reading the amount 1,250.00 as 12,500.00. The same goes for a missing minus sign or confusing the debit and credit columns. Even with a high average score, critical errors in amounts, identifiers, signs, periods or entity links can be hidden. Human review also cannot be assumed to be free or perfect.

Method

A financial fact is defined as a typed record linked to a visible region of the source document. Verbatim transcription, deterministic normalization and business interpretation are recorded separately. Quality is treated not as a single score but as a vector of 10 layers: detection, text, layout, tables, fields, factual accuracy, evidence linking, uncertainty calibration, robustness and operational performance. The critical-field error rate is subject to a non-compensatory gate that uses a one-sided upper confidence bound. The aim is to prevent high scores on other items from offsetting critical errors. Selective automation is formulated as a constrained optimization problem that accounts for residual risk, reviewer capacity and review cost. The paper also presents annotation and adjudication procedures, data splits that prevent information leakage, subgroup analysis, reproducibility cards and a design for an operational control plane.

Results

• There are no new models, datasets or experimental performance results; the authors state that the contribution is theoretical and methodological.
• The paper formally showed the limitation that compensatory (weighted-average) scores can hide failures in critical fields.
• It showed that even when zero errors are observed, the upper confidence bound on risk is not zero. On this basis, the authors argued that pass decisions must account for sample size.
• The paper derived a loss-based decision rule for sending uncertain cases to human review. It also presented evaluation priorities for 9 categories of financial documents, such as bank statements and invoices, and 4 synthetic example documents.

Limitations

This is a purely methodological paper with no experimental validation. Weights, criticality grades, tolerances and pass criteria are left as risk decisions that the adopting organization must set in advance. The 4 synthetic documents are for illustration only and are not benchmark data. The authors state that the framework's scores do not guarantee regulatory compliance or accuracy on future documents.

Why it matters

In document automation at banks and companies, real risk comes less from average accuracy than from a small number of critical errors in amounts, signs and identifiers. This framework can serve as a checklist for designing pass criteria, the allocation of human review and audit records when adopting OCR or document AI.

Key terms

CER / WER

A recognition error rate computed from edit distance at the character or word level. Its limitation is that it weighs every error equally.

Evidence-grounded annotation

An approach that records, for each extracted value, its location in the source image, its verbatim transcription, its normalized value, its semantic role and whether it is illegible.

Non-compensatory gate

An evaluation rule that fails a system if its critical-field error rate exceeds the threshold, no matter how high its scores are on other items.

Selective automation

An operating approach that automatically processes only cases with sufficient confidence and sends uncertain cases to human review. It takes both review cost and reviewer capacity into account.

Original

• arXiv abstract
• PDF
• HTML full text
• Hugging Face

Authors: Subodh Lonkar · Posted 2026-11-02 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.