Paper review · October 8, 2026
From Evidence to Action: Where Do Tool-Using Agents Fail?
From Evidence to Action: How Tool-Using Agents Fail
The authors built SafeActBench, a benchmark that tracks whether agents actually obtain the evidence they need before acting. It shows that most failures of tool-using agents begin with insufficient investigation and premature action before execution.
Problem
Tool-using agents take actions that change external state, such as issuing refunds or editing records. But even when the final outcome is correct, there is no guarantee that the action was based on evidence gathered beforehand. Existing evaluations mostly look only at task completion or the final state, so it is hard to tell where the chain from evidence to action breaks.
Method
SafeActBench was built with 656 cases across 6 business domains. There are 5 protocols: static judgment (Legacy, Allow/Block/Defer), investigated non-action (V0), single action (V1), linear multi-action (V2) and dependency-graph (DAG) multi-action (V3). A provenance-based Evidence Ledger records, for each fact, which interaction established it and for which entity and state. For example, a different claim with the same amount is not accepted as evidence for the target claim. A deterministic evaluator replays the trajectory and checks investigation completeness, correctness of tool, target and arguments, use of execution results and dependency order. No LLM judging is used. Five model families were evaluated in 10 configurations, each paired with its own dedicated harness and with a common ReAct harness (Inspect).
Results
- The best configuration was Claude-5 + Claude Code, with an overall exact correct success rate (ECS) of 67.2%. The lowest was GLM-5.2 + ZCode at 37.7%.
- High static judgment performance did not guarantee strong interactive execution. GLM–ZCode scored 97.7% on Legacy but 12.1% on V2. Three configurations re-evaluated on the same V1 cases also had static accuracy of 95% or higher but interactive ECS of 52% or lower.
- Harness effects were large and varied by model. DeepSeek-V4 rose from 57.5% with Inspect to 63.0% with DSH. For GLM-5.2, Inspect (41.0%) beat ZCode (37.7%). For the other families, the dedicated harness was 2.1–4.4 percentage points higher.
- The authors report that failures mostly begin before execution: agents stop after incomplete investigation, or act before obtaining the necessary evidence. Single-action execution after the evidence was obtained was generally stable. Multi-action settings additionally revealed unresolved preconditions and incomplete execution.
Limitations
The evaluation covered 656 cases in 6 domains in simulated tool environments. The authors note that because the evaluation only looks at observable trajectories, it cannot determine what the model internally relied on. Evidence requirements depend on case specifications defined by the authors. There are 10 evaluated model and harness configurations.
Why it matters
When agents are deployed for real work, what matters is not only whether the result is correct but whether they checked the evidence before acting. This study provides diagnostic criteria showing that the investigation stage and dependency handling need to be checked separately. That the choice of harness strongly affects performance is also a practical takeaway.
Key terms
- Consequential action
- A tool call that changes application state or has external effects, such as issuing a refund, sending a document or editing a record. It is distinct from information-lookup calls.
- Investigated non-action
- Not simply refusing in a situation where the agent should not act. The agent must complete the necessary investigation, confirm the reason, and then stop without acting.
- Evidence Ledger
- A ledger that records each established fact tied to the interaction it came from and the target entity and state. It is used to judge whether the necessary evidence was actually obtained right before an action.
- Harness
- An agent execution framework that wraps a model and manages tool calls, iteration loops and prompt construction. Examples include Claude Code, Codex and ReAct implementations.
Original
Authors: Hongzhan Lin, Shidong Cao, Ziyang Luo, Wenhao Chai, Mong-Li Lee, Wynne Hsu · Posted 2026-10-06 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.