Paper review · October 28, 2026
Trial by Trace: A Benchmark for Evaluating the Reliability and Failure Causes of AI Agent Harnesses
Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution
The authors built a 95-task, two-layer evaluation framework that separates agent performance into model and harness, revealing harness defects hidden behind task success.
Problem
Agent performance depends not only on the model but also on the harness: context management, tool delivery, permissions, session management and so on. Existing benchmarks look only at the final answer or whether tests pass, so harness defects such as lost context or incorrectly delivered tool results are masked by task success, and when a task fails it is impossible to tell whether the model or the harness is at fault.
Method
The authors split harness reliability into seven components (40 sub-properties)—Context, Execution, Governance, Lifecycle, Observability, Tools and Verification—and created 95 expert-level tasks that each exercise specific mechanisms. Each task has an executable or state-based ground-truth criterion. Codex, Claude Code, OpenClaw and DeepSeek Harness are run through adapters while keeping their native behavior. Trajectory Tracing (TT) reconstructs execution records into a common format (Trace IR), the MEA module gathers evidence from the agent context, harness and environment state, and an LLM judge scores the results. Agent-layer and harness-layer scores are assigned separately, and the Failure Attribution (FA) module uses predefined mechanism chains to find candidate links between harness failures and agent failures.
Results
- A total of 3,420 runs were performed across 4 harnesses, 9 models and 95 tasks.
- Macro-average scores per harness varied from 59.8% to 72.0%.
- 31.1% of all runs succeeded at the task but failed at least one harness criterion.
- The authors report that harness rankings differed depending on whether they were based on task success rate or on harness-criteria pass rate.
Limitations
With 95 tasks, the task set is small and unevenly distributed across components (27 for Execution, 1 for Verification, 7 for Observability). Only one selected run per model–task pair is counted. An LLM judge is used for semantic judgments, and the authors note that automatic attribution only proposes "candidate links"; confirming actual failure propagation requires manual review.
Why it matters
As the same model is increasingly used through multiple harnesses, as with coding agents, harness choice and improvement drive real-world quality. This benchmark provides an auditable evaluation framework that flags runs where "the task succeeded but something went wrong internally" and traces the cause. It can serve as a reference when designing agent operations and observability systems.
Key terms
- Agent Harness
- The execution system that wraps a model and handles context construction, tool exposure and calls, state persistence, permission control and so on. The authors define Agent = Model + Harness.
- Trace IR
- An intermediate representation that normalizes each harness's different execution records into a common format while preserving provenance information.
- Failure Attribution
- The process of estimating which harness defect caused a failed task outcome by linking them through predefined mechanism chains.
Original
Authors: Zhiyong Cao, Hengjie Zhu, Yongxin Wang, Xuchen Li, Jintao Li, Shuxia Liu, Dunqiang Liu, Bo Wang, et al. · Posted 2026-10-23 · License CC BY-NC-SA 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.