Coding agents and SWE-bench: solving problems at the repository level

Why it was needed: the gap between 'writing code' and 'software engineering'

When large language models (LLMs) first showed coding ability, the main question tested was 'given a description, can it write a single function?' But most of a real developer's day is spent reading a large codebase written by others, finding where a bug comes from, making small changes across several files, and then checking that existing features still work. Function-level evaluation cannot capture these abilities.

The USEagent study paper took up this gap directly. Its point is that software engineering covers not just coding but also maintaining and evolving a project. So instead of separate specialized agents built for test writing, debugging or repair, it aimed for a single agent that handles situations such as finishing an incomplete patch, adding a new feature, or taking over code someone else wrote.

In this context, repository-level evaluation became central. It takes real repositories and real issues as they are and judges the agent's fix with tests. SWE-bench became the representative benchmark (a test set for comparing performance) for this approach, and SWE-Bench Pro paper kept SWE-bench's good practices while extending the scope to harder problems closer to enterprise work.

Core idea: an exam that hands an issue ticket to a new developer

Repository-level evaluation resembles an exam where a newly hired developer is given an unfamiliar project and a single issue ticket and told to 'fix it'. The grader does not judge the code by eye. Instead, it runs tests prepared in advance and hidden from the candidate, checking whether the originally broken behavior is fixed and whether features that worked before still work.

What makes this exam hard is that the 'shape' of the correct code is not fixed. The candidate decides which files to change and how many lines to modify. So, alongside the ability to write code, 'finding where to look', 'the habit of running things to check them' and 'judging whether a fix breaks something elsewhere' become important.

The analogy also shows the weak points. If the exam paper leaks, the ticket description is wrong, or the tests check something other than the issue, the score does not reflect real ability. SWE-Bench Pro Verified paper, covered later, pointed out exactly this problem.

How it works: from issue to patch to grading

First, there are two inputs: the issue description (what is wrong and what is wanted) and a snapshot of the repository at the time the issue was filed. The agent works inside a sandbox (an execution environment isolated from the outside) where this repository is installed.

Second, the agent works through many back-and-forth steps. It looks at the file list, searches for related code, opens and reads suspicious parts, edits them, and runs tests or reproduction scripts to observe the results. The full record of this 'think → use tool → observe' sequence is called a Trajectory. The prompts, tool list, execution environment and context-management mechanisms that wrap the model so it can use these tools are called the harness.

Third, when the agent declares it is done, the changes to the repository are extracted as a patch. SWE-Bench Pro paper includes long-horizon tasks that could take a professional developer hours to days, and reports that many require patches spanning multiple files and substantial code changes.

Fourth, grading uses hidden tests. The patch is applied to the original repository, the tests are run, and if they pass the task counts as 'resolved'. Because correctness is decided by execution results rather than human judgment, the same structure can be used directly as a reward for reinforcement learning. This is called reinforcement learning with verifiable rewards (RLVR).

How it developed: harder, and more trustworthy

SWE-Bench Pro paper built on SWE-bench to include complex, enterprise-level problems. It collected 1,865 problems from 41 repositories spanning business applications, B2B services and developer tools. Its distinguishing feature is how the sets are organized: a public set from 11 repositories, a held-out set from 12 repositories, and a commercial set from 18 proprietary repositories obtained through agreements with early-stage startups. Drawing problems from unpublished code reduces the chance that a model saw the answers during training, that is, data contamination. Every task was verified by humans and supplemented with enough context to be solvable. The authors also grouped failure types in the collected Trajectories to analyze how current models go wrong.

The USEagent study paper broadened the scope in a different direction. To see whether one agent can handle several kinds of work, such as coding, testing and patching, rather than just issue repair, it built USEbench by combining tasks from SWE-bench, SWT-bench and REPOCOD. It evaluated on 1,271 repository-level tasks.

SWE-Bench Pro Verified paper asked, 'Can SWE-Bench Pro scores be trusted?' It found two reliability problems. One is reward hacking, where reference solutions or hidden evaluation information leak and are exploited. The other is task quality, such as misleading problem descriptions or tests with the wrong scope. The authors added safeguards that block the main leakage paths without impairing normal agent functions, and made minimal fixes to flawed tasks. On the verified version, some models scored much lower than in the earlier evaluation, and the researchers concluded that the original SWE-Bench Pro results may have overestimated real software engineering ability.

Where it is heading: harnesses, long-horizon reinforcement learning, verification cost

First, the view that 'looking at the model alone is not enough'. The harness effects study paper compared two harnesses (mini-SWE-agent, OpenCode) and 10 open-weight models from the Qwen and DeepSeek families on three benchmarks: SWE-bench Pro, ProgramBench and GitTaskBench. The effect of the harness depended on both model capability and task type. For SWE-style issue repair, the extra benefit of a complex harness shrank as models got stronger, but on more complex, open-ended repository tasks even strong models could benefit from a complex harness. By component, structured tool use and task-specific subagents helped most consistently, while context compression and general-purpose subagents sometimes lowered repository generation performance.

Second, methods for teaching long tasks with reinforcement learning. In long-horizon tasks a single small mistake can ruin an entire Rollout, so successful Trajectories are rare. Restarting from scratch is inefficient, and rolling back only the conversation context and working folder misses environment changes such as installed dependencies. SPROUT paper checkpoints the file system and agent context together at every turn, and uses a verifier-guided backtracking policy to choose where to resume a failed attempt. On long-horizon coding tasks it reported up to 23.6% improvement over pass@k (a metric that counts a task as solved if at least one of k attempts succeeds) while using only 32–52% of the tokens. In training Qwen3.8-27B, it scored 13.9 points higher than standard GRPO across three benchmarks and cut time per training step by 39.5%.

Third, the cost of test-time compute. Generating several candidate Trajectories and picking the best one improves performance, but verifying long records can cost as many tokens as generating them. Instead of a second read by an LLM verifier, LatentSift paper filters candidates by comparing the Hidden Representations the policy model already produced during generation against successful and failed examples. On SWE-bench Verified with K=16, it reduced execution-free verifier tokens by 66.6–81.0% and total verification tokens, including test generation, by 49.1–62.1%; on DeepSWE-Preview, Best@16 rose from 59.26% to 60.06%.

Limitations and open issues: what scores tell you, and what they don't

The first point is benchmark reliability. As SWE-Bench Pro Verified paper showed, solution leakage and task defects inflate scores. Even held-out sets meant to prevent data contamination paper do not solve the problem if the tests themselves do not properly check the issue. When looking at leaderboard numbers, first check which set was used and whether leakage blocking and task verification were applied.

Second, the unit of comparison. According to the harness effects study paper, performance is not determined by the base model alone. The same model can give different results with a different harness, and the effect can even change direction depending on task type. The number of attempts also matters. Scores that pick from multiple attempts, such as pass@k or Best@16, do not mean the same thing as a score from a single attempt.

Third, work outside the benchmark. The spec-driven development study paper summarized industry reports of a 'productivity paradox', where individual productivity rises but team throughput, review capacity and stability decline, and argued that teams need specifications that act as a contract between people and agents, along with harnesses that control agent behavior. MAGE paper likewise found that coding agents increase implementation capacity but do not automatically surface a project's intent, system architecture or acceptance rationale. Fixing a single issue so the tests pass and judging what should be accepted are different problems.

Further study

A good starting point is SWE-Bench Pro paper. In one paper it covers how repository-level tasks are built, why public, held-out and commercial sets are separated, and how agents fail. Reading SWE-Bench Pro Verified paper next lets you compare how scores on the same benchmark can be inflated.

If you are building or choosing an agent, the harness effects study paper is closest to practice. It turns components such as the tool list, context compression, planning and subagents on and off one at a time to measure their effects, which helps decide which mechanisms to add first. To look at work beyond issue repair, read USEagent paper alongside it.

If you are interested in model training, see SPROUT paper for the bottlenecks of long-horizon reinforcement learning and LatentSift paper for the cost of verifying and selecting among multiple candidates. The spec-driven development study paper and MAGE paper cover the team and organizational perspective.