Paper review · October 20, 2026
BizBench: A benchmark that exposes hidden challenges in deploying long-horizon agents in real business environments
BizBench: Unveiling Hidden Challenges in Deploying Long-Horizon Agents for Real-World Business
Based on a payments company's real business procedures, the authors built a benchmark that checks not only final results but also compliance during intermediate execution. Even frontier models reached only about 60% on the strict success rate.
Problem
Existing long-horizon agent benchmarks focus on whether the final task succeeds, which makes it hard to catch violations in intermediate steps. In high-risk work such as payments, a transfer without approval or a minimum-balance violation counts as a failure in itself, even if the result is corrected later. Real business rules are also scattered across documents that are outdated or conflict with each other.
Method
The authors organized a global payments company's business documents into SOPs, rewrote sensitive content, and rebuilt them as an executable environment. The environment contains 789 knowledge documents and 601 business tools that agents must search for and find before using. The 50 tasks include resource constraints, time dependencies, prior commitments, and outdated or conflicting guidelines. An LLM user simulator provides customer information and approvals. Evaluation uses deterministic checks that count a run as successful only if it meets all of goal achievement (G), requirement compliance during execution (P), and protected items and forbidden side effects (F); no LLM judge is used. The evaluator was validated by replaying golden traces and with deliberately constructed violation cases.
Results
- The authors report strict success rates of only 64.0% for Claude Opus 5 and 61.3% for GPT-6 Sol.
- Even GPT-6 Sol succeeded on all three attempts for only 40.0% of tasks, so results varied a lot from run to run.
- Qwen3.8-27B had a goal achievement rate of 42.0%, but its strict success rate, which also accounts for violations during execution, dropped to 32.7%. This shows that outcome-only evaluation can hide faulty execution.
- DeepSeek-V4.1-Flash made an average of 191.3 tool calls per run, about 70% more than the reference run. Qwen3.5-9B did not achieve all goals in a single one of its 150 runs.
Limitations
There are 50 tasks, based on the payments operations of a single company. Business details were abstracted and rewritten to protect confidential information, so the benchmark may differ from the real environment. An LLM simulator plays the user, so the simulator's behavior may affect results.
Why it matters
When deploying agents in financial and enterprise work, "Did the process follow the rules?" matters as much as "Is the result correct?" This benchmark shows that difference quantitatively and shows how to build practical checks, such as approval order, minimum balances and identifying which rules are valid, into evaluation design.
Key terms
- Strict success rate (Strict Success)
- A metric that counts a run as successful only if it achieves the final goal and also complies with approvals and constraints during execution and avoids forbidden side effects.
- Golden Trace
- A reference execution record that solves the task correctly. It is used to verify that the goals and constraints can be satisfied together, and does not force a single correct path.
- User simulator
- A component in which an LLM plays the customer, providing customer information during the conversation and responding to approval requests.
Original
Authors: Enhan Zhao, Yongqi Tong, Xuejun Yu, Hang Wang, Pan Wang, Yichao Wang, Di He, Jianshe Li, et al. · Posted 2026-10-14 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.