Harness engineering: designing what sits outside the model to make agents reliable

Why it was needed: problems that changing the model alone does not solve

Large language models (LLMs) are now used for more than chatbots that finish with a single answer. They are also used for work that calls tools, edits real files and carries on across many steps. Doc2Harness paper points out that in such long-running tasks, an agent's performance depends as much on the harness around the model as on the model itself. A harness (agent harness) is the system layer outside the model that bundles prompts, tools, memory and orchestration logic (the category-theoretic architecture study paper). Orchestration is the control that decides in what order multiple steps and agents run.

SemaClaw paper treats early 2026, when OpenClaw appeared and millions of people began using personal AI agents in daily life, as a turning point. Its claim is that from then on, the focus of AI engineering has been shifting from prompt engineering and context engineering to harness engineering. The study argues that as model capabilities converge, the harness layer becomes where products differ from one another.

The problem is that this layer was long a hand-tuned craft. LLM agents behave differently on every run, even given the same task and the same tools. The determinism-constraints study paper notes that this variance may be fine for exploration but is unacceptable in regulated domains such as finance and compliance. The category-theoretic architecture study paper considers harness design ad hoc: there is no systematic theory for the principles of combining components or for comparing frameworks with one another.

Core idea: tack that fits a strong horse

The English word harness originally means the tack put on a horse. A strong horse can run on its own, but it needs reins, a yoke and a cart to carry a load in the direction you want. In this analogy the LLM is the horse, and the harness is the full set of tack that turns its strength into work. Harness engineering is the work of designing this tack.

Comparing it with prompts and context makes the difference clear. Prompt engineering deals with how to pull the reins, that is, the wording of instructions. Context engineering deals with what to show in the horse's field of view, that is, the information placed in the context window. Harness engineering goes further and also decides which tools to permit, who checks the results, how many times to retry on failure and where to keep task state. SemaClaw paper describes this as designing the full infrastructure that turns an unconstrained agent into a system that is controllable, auditable and reliable in production.

The category-theoretic architecture study paper organizes the capabilities an agent has moved outside the model into four pillars: memory, skills, protocols and harness engineering. It treats the full harness as the 'architecture' that ties these together. In other words, a harness is not a single component but the way components are put together.

How it works: the course of one turn

The flow by which a harness handles one turn is roughly as follows. It combines components from several studies in a typical order, and not every harness follows this exact order.

First, it builds the context. It selects the system prompt, the skills needed for the current task (bundles of reusable task instructions) and records of previous work, and feeds them to the model. SemaClaw paper uses a three-tier context management structure for this. Second, the model proposes the next action: whether to call a tool, and with what arguments. Third, the harness checks that proposal instead of running it right away. SemaClaw's PermissionBridge is a permission layer responsible for action safety. The determinism-constraints study paper builds a harness from five components: finite-state control that allows moves only between defined states, forced tool selection, output validation, a cap on retries and structured planning.

Fourth, actions that pass the check are run as tools in an isolated environment such as a sandbox, and the results are saved to a state store. Fifth, those results are fed back into the context and the next turn begins. NORA paper names five harness elements for research agents: lifecycle hooks that run automatically at specific points, safety gates, separation of generator and evaluator, human-in-the-loop and state persistence. The study argues that these elements make reliable, reproducible autonomous research possible.

The key to this structure is the division of roles: 'the model decides; the harness permits, verifies and records.' Even if the model outputs a wrong action, the harness can filter it out or have it retried. It can also narrow the range so that the same input follows the same path.

How it developed: from building to automation and measurement

Starting in spring 2026, a series of studies with 'harness engineering' in their titles appeared. The first strand was building and demonstrating real harnesses. SemaClaw paper is an open-source multi-agent framework. It includes DAG-based (a graph whose arrows link tasks by their order of dependency) two-stage hybrid agent-team orchestration, PermissionBridge, three-tier context management and a wiki skill that automatically builds a personal knowledge base. NORA paper runs the full process of spatial data science research with 21 domain skills, 9 specialized subagents and a custom MCP server.

The second strand is automation and theory that take the harness out of human hands. AHE (Agentic Harness Engineering, paper) gives three reasons automation is hard: the editable components are heterogeneous, trajectories (records of an agent's actions) are so large that useful signals get buried, and it is hard to tell which edit had an effect. AHE adds three mechanisms to match. It gives each component a revertible file-level representation and distills records of millions of tokens into evidence bundles that can be drilled into layer by layer. It also attaches an expected effect to every edit and verifies it against the next round's results. As a result, it reports raising pass@1 (the share solved on the first attempt) on Terminal-Bench 2 from 69.7% to 77.0% over 10 iterations, surpassing the human-designed Codex-CLI harness (71.9%). The category-theoretic architecture study paper formalized the harness as a mathematical structure. It also showed that compilers targeting Swarms, DeerFlow, Ralph, Scion and LangGraph preserve three kinds of guarantee certificates.

The third strand is measuring the harness itself. The determinism-constraints study paper measured how much a harness changes run reproducibility. LoopsBench paper holds that the focus of coding-agent infrastructure is moving from harness engineering to 'loop engineering' (designing long-running development iterations), and evaluates with 112 long-horizon tasks. Doc2Harness paper differs from earlier benchmarks that measured only task success on top of a fixed harness. It has models build the harness themselves and then evaluates the result. Among 19 models, the best, GPT-6 Astra, passed 53.05% of 328 test cases, and the lowest model scored 27.44%.

Where it is heading now

Recent research asks how much, and how, the harness matters along three lines. First, how much does a harness close the gap between models? An industrial pipeline study that converts US state bills into strict-schema XML paper takes up this question. It crossed three harnesses (zero-shot, a single agent with tools, and a decomposed multi-agent pipeline) with three Claude models and compared them on 153 gold-standard bills. The weakest model placed in the full pipeline beat the strongest model used on its own (0.948 vs. 0.806). Because the two agentic harnesses had the same tools, the authors attribute the difference to orchestration. Their explanation is that splitting the work preserved state and gave each agent its own context. The errors that remained at the end were concentrated in citation normalization. There, a deterministic tool that looks up canonical identifiers from an external authority, instead of having the model generate them, filled the accuracy gap.

The second is the opposite question: is a complex harness really needed? OpenRUA paper did not build a custom abstraction layer for its robot agent. Instead, it chose a 'the workspace is the harness' design that gives an off-the-shelf coding agent only terminal access to ROS 2, the robot's basic software interface, along with documentation and basic tools. Using Claude Code based on Claude Opus 5, it reported success rates of 99.0% on CaP-Bench and 87.0% on LIBERO-PRO. This result shows it can work without custom components or task-specific training.

The third is theory that treats harness evolution as a learning problem. The study on harness evolution for personal agents paper covers a setting where the model stays fixed and only the harness that manages context, memory, tools and execution is changed to fit user preferences. It explains the limits of this process in terms of approximation, generalization and optimization error. It is an attempt to work out the range of behavior reachable through harness edits alone and the limits set by limited interaction evidence.

Limitations and open issues

Adding more harness does not always help. The determinism-constraints study paper built four cells by pairing two models, Qwen-2.5-7B-Instruct and Gemma-3-27B, with two synthetic tasks, finance and legal. The first harness clearly improved reproducibility in only one cell, actually reduced it in two, and made no difference in one. The cause was the planning step written in free text. Once tool order, state order and output were already consistent, this step was the main remaining source of variance. After adding structured planning, which validates the plan against a defined schema before any tool call, three of the four cells reached 1.000 on the reproducibility and determinism metrics over 100 runs. There were costs, too. Token cost fell in every cell, but latency dropped for one model and rose noticeably for the other.

Harness thickness is also debated. An approach that stacks many components, like NORA paper and SemaClaw paper, is used alongside one that provides only the minimum, like OpenRUA paper. The bill conversion study paper and OpenRUA cover different domains. So rather than declaring one side right, it fits the evidence better to say that the answer depends on the accuracy the task demands and how well the model handles existing interfaces.

Building a harness is itself still hard. In Doc2Harness paper, even the strongest model failed nearly half of the test cases. Failures clustered in splitting up functionality that spans multiple modules. In LoopsBench paper, even the strongest configuration (Opus-4.7, Claude Code, an outer continuation loop) solved only 25.00% of tasks. The plans agents made recovered only part of the actual dependency order, and regressions also appeared, where already finished parts broke again.

The scope of formal guarantees also needs to be read precisely. The compiler in the category-theoretic architecture study paper checks whether structure is preserved and checks verifier re-execution. The authors state directly that it does not check whether outputs are correct or how the model behaves. What the harness guarantees and what the model can still get wrong need to be considered separately.

Further reading

If you are new to the topic, start with SemaClaw paper. It gives a sense of why harness engineering emerged and what components such as permissions and context management look like. Next, NORA paper shows how hooks, gates and generator–evaluator separation are used in a real research agent.

To see design effects in numbers, read the determinism-constraints study paper and the bill-to-XML conversion study paper in turn. Through controlled experiments, the first shows that a harness can backfire, and the second shows that orchestration makes a difference even with the same tools. For automatically improving harnesses, start with AHE paper; for evaluating harnesses, start with Doc2Harness paper and LoopsBench paper.

If you are interested in theory, read the category-theoretic architecture study paper and the harness evolution study paper. Reading OpenRUA paper, which takes the opposite view, alongside them gives a balanced view of the question 'how much harness should you build?'