Paper review · October 16, 2026
Doc2Harness: An LLM Benchmark for Measuring Coding Agents' Long-Horizon Harness Engineering Ability
Doc2Harness: Benchmarking LLMs on Long-Horizon Harness Engineering of Coding Agents
It has LLMs build an agent harness themselves from development docs and an interface-only skeleton, then scores deterministically with a mock model to measure harness engineering ability.
Problem
A coding agent's performance depends as much on the harness that manages tool execution, context, permissions and session state as on the model. Yet existing benchmarks fix the harness and measure only final task success. They do not measure how well an LLM can build the harness itself. In addition, the same feature can be implemented in many ways, and a harness only runs with a model attached, which makes it hard to evaluate with ordinary unit tests.
Method
In stage 1, the model under evaluation works inside a fixed builder harness (OpenCode). The model receives development docs that consolidate and organize the user documentation of 12 recent harnesses, plus an interface-only contract skeleton (based on mini-swe-agent), and completes the harness in an isolated container. In stage 2, the completed harness is placed in a separate container for each test case and run end to end with a mock model that responds according to a script. Scoring uses four channels: requests sent to the model, files on disk, exit codes and command output. Cases that pass even though the feature did not actually run are filtered out with positive controls. Test cases were built from an analysis of more than 20,000 public issues, comparisons with existing harnesses, and extensions that change only one variable at a time.
Results
- 19 recent models were evaluated with 3 independent builds per model and 328 test cases. GPT-6 Astra had the highest pass rate at 53.05%, and the lowest model scored 27.44% (a gap of about 26 pp).
- Even the strongest model failed nearly half of the test cases. The authors report that failures were concentrated in how features spanning multiple modules were split up.
- Higher-scoring harnesses tended to consolidate features spanning multiple modules into fewer modules.
- Across the six main categories, the same model's rank varied by up to 16 places.
Limitations
The builder harness is fixed to OpenCode alone, so results may differ in other builder environments. Because the mock model responds according to a fixed script, quality when interacting with a real LLM is not measured directly. The contract skeleton comes from the Python-based mini-swe-agent, and the test cases were designed by the authors. The provided text was cut off, so only part of the detailed analysis was checked.
Why it matters
It is the first to deterministically measure the ability to build a harness, which would be central in self-improvement scenarios where agents modify their own infrastructure. It serves as a reference for model selection for teams building or choosing coding agents.
Key terms
- Harness
- An agent execution framework that wraps an LLM and manages the interaction loop, tool execution, context, permissions, session state and error recovery. Examples include Claude Code and Codex CLI.
- Mock Model
- A fake model that returns responses according to a predefined script instead of a real LLM. It makes evaluation deterministic and reproducible.
- Vacuous Pass
- A case where the feature under test never actually ran, but an assertion happens to be true and the test passes.
- Positive Control
- An auxiliary test that checks an outcome that must occur in the same scenario, verifying that the observation's preconditions hold. If it fails, the original test is also marked FAIL.
Original
Authors: Yuyao Ge, Shenghua Liu, Peizhong Gao, Chendi Ge, Chenxu Liu, Yiwei Wang, Lingrui Mei, Baolong Bi, et al. · Posted 2026-10-14 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.