AI Agent Harnesses: What a 447-Task Study Shows
A 447-task study finds harness changes can match rerun noise. Learn how to benchmark agent wrappers, interpret score shifts, and compare costs.
When an AI Agent Harness Changes, Is the Score Gain Real?
Changing an AI agent's harness can look like a clean upgrade: swap in a different system prompt, add typed tools, tweak context handling, then compare benchmark scores. The snag is that agent runs aren't perfectly repeatable. Even with the same model and setup, a task that passes once can fail the next time through. If you don't measure that "same config" variation first, it's easy to mistake run-to-run noise for a real harness effect.
A preprint study of 447 SWE-bench Verified tasks gives a concrete number for how often that happens. Rerunning the same configuration changed the outcome on 13% of tasks at the median. Switching between harnesses also changed outcomes on 13%. The study isn't the last word, but it offers a practical way to read agent benchmarks: separate run-to-run noise from wrapper effects, and compare cost as well as pass rate.
This article walks through what the study tested, what its results do and don't show, and how to evaluate harness changes in your own workflow.
What Did the 447-Task Study Test?
The preprint, "What Does a Harness Buy? Tokens, Mostly," by Yangze Liu and Zhongyi Han of Shandong University was published on arXiv on October 3, 2026. It compares three harnesses around language models on SWE-bench Verified, a benchmark built from real GitHub issues and associated tests. Since the paper hasn't been peer reviewed, treat it as useful evidence - not settled guidance.
A harness is the operating layer around a model. It can include the system prompt, available tools, context management, output limits, and rules for continuing or retrying a task. In the study, the harnesses were:
- Claude Code: a heavier setup, with a longer system prompt, typed file and search tools, and automatic context compaction.
- mini-SWE-agent: deliberately lean, using a single bash tool and very little context management.
- OpenCode: in between, with typed tools and a smaller preamble.
The researchers tested five models, but not every model ran across every harness and task pool. After excluding eight tasks that required external services, 492 tasks remained. A 447-task pool was used for two models; a separate 45-task hard subset was used with all five. Claude Opus 5 appeared only within Claude Code as a ceiling reference, not as a direct cross-harness comparison.
The key methodological move was the control group: ten pairs of runs with the same model, harness, tasks, and settings. That's what lets you estimate how much results move even when you don't change anything. Without that baseline, a reported harness difference of a few percentage points is hard to trust - it could just be ordinary run-to-run variation.
This is the study's most practical contribution: it asks whether the observed harness change is bigger than the system's own instability.
How Much of a Harness Result Is Real?
At the task level, the study found that reruns changed outcomes on 13% of tasks at the median. Switching harnesses changed outcomes on 13% too. Switching models changed outcomes on 22% of tasks.
One important detail: these are task-level flip rates, not score improvements. A flip means a task that passed in one run failed in another (or vice versa). Since flips can go either direction, a 13% flip rate does not mean the overall benchmark score shifts by 13 points. What it does mean is that a meaningful share of individual task outcomes is unstable or sensitive to the configuration you're comparing.
That matters when you're dealing with close results. If you run harness A once and harness B once, you can't assume every score difference comes from the harness. Some of the "changed" tasks may have flipped even if you'd rerun harness A unchanged.
What the score comparison actually supports
On the 447-task pool, Claude Code and mini-SWE-agent - the heaviest and lightest setups - were within five points of each other for both tested models. The measured gaps were -1.8 and -1.4 points, with 90% intervals staying within plus or minus five points. Put simply: on that pool, the study didn't show a reliable advantage for either harness.
OpenCode was the notable exception: it trailed by as much as nine points. For one model, about half of that gap came from runs that were cut short by the output cap. Claude Code prompted the model to continue when its reasoning ran long; OpenCode ended the step. The authors tried adding Claude Code's continuation prompt to OpenCode, but didn't reliably recover the difference. The same overflow didn't consistently recur on a retry.
So this doesn't just read like "Harness X beats Harness Y." It's also a reminder that harness results can hinge on specific failure modes - output limits, retry behavior, and context handling - not just general "quality."
Why small benchmarks miss small effects
The study's estimates suggest that 45 tasks detect a 13-point gap only about half the time. To resolve a five-point difference under the observed flip rates, you need roughly 447 tasks. That means a claimed 2.4-point lead is sitting below what either sample size can confidently separate from noise.
Exact requirements will vary with your tasks and your agent, of course. But the general rule holds: the smaller the improvement you expect, the more runs and representative tasks you need to tell it apart from instability. A single benchmark pass is a shaky foundation for a migration decision when the expected gain is small.
Why Does the Harness Still Matter?
This study doesn't argue that harness engineering is pointless. It shows something narrower: a harness change may not deliver a dependable pass-rate advantage in a specific comparison, while still changing cost, reliability, and how an agent fails.
Cost was the clearest difference. With the same model and tasks, the cost per task varied by as much as three times across harnesses under the paper's price assumptions. A big driver was the fixed preamble: the system prompt and tool schemas sent with each model step. Claude Code used about 16,000 tokens, OpenCode about 7,000, and mini-SWE-agent fewer than 1,000.
A fixed preamble gets paid repeatedly. If a task takes many steps, the harness resends that prompt and tool definition as part of the interaction. The study found that per-call tool output and per-step growth differed less than the preamble did. In other words: once a task has enough steps, a large prompt can dominate the bill.
Measure cost per solved task, not just tokens per run
A cheaper run isn't automatically a better system. If a lean harness fails more often, you may pay less in tokens and still lose on cost per successful task. On the other hand, a larger prompt and more capable context management might not be worth it if the pass rate stays basically tied.
For a real deployment, track at least:
- Cost per attempted task, including retries and failed runs.
- Cost per solved task, using a consistent definition of "solved."
- Task completion rate, measured across repeated runs.
- Failure categories, such as timeouts, output truncation, tool errors, or incorrect patches.
- Latency, especially if the agent is sitting inside a developer workflow where waiting time affects adoption.
That gives you a decision metric that reflects both quality and operating expense. It also makes trade-offs visible: a more expensive harness can still be the right call if it consistently prevents expensive failures or reduces human review. But you have to see that benefit on your task mix - not just in a polished comparison.
Reliability is often a more useful target than a higher average
A harness can improve robustness without moving a headline benchmark score much. Better continuation behavior might prevent a task from getting cut off. Context compaction might help the agent handle longer issues. More reliable tool calls might reduce hard failures.
Those improvements matter in production, where one stalled task can interrupt a developer, force a retry, or require manual repair. A benchmark score can hide those operational consequences. The real question isn't whether the harness is "better" in the abstract. It's whether its behavior improves what you actually pay for.
How Should Teams Test an Agent Harness?
A practical lesson from the study is to treat harness evaluation like a controlled experiment. A comparison is only useful when you can attribute the difference to one variable and estimate how much the result shifts even without changing anything.
1. Establish your own noise floor
Run the same model, harness, task set, and settings more than once. Compare outcomes task by task, not only the aggregate score. If a meaningful share of tasks flips between identical runs, that's your baseline instability.
Keep task definitions and scoring rules the same each time. Record model version, system prompt, tool configuration, limits, and retry settings. If any of those drift between runs, you're no longer measuring repeatability of the "same config" - you're measuring changes on top of changes.
Also: your internal workload matters more than a published benchmark for this step. A coding team fixing small bugs may see a different variance profile than a team doing broad refactors or repo migrations. Build a representative set from the work you actually expect the agent to do.
2. Change one thing at a time
To estimate a harness effect, hold the model constant and change the harness. To estimate a model effect, hold the harness constant and change the model. If both change at once, you can't tell what drove the outcome.
That sounds obvious, but production evaluations often bundle a new model with a new wrapper, different limits, and revised prompts in one release. That can be fine as a product experiment. It's just not a clean harness comparison. If you're disappointed, you won't know why; if you're happy, you also won't know what part earned the gain.
3. Make the sample match the decision
Pick a sample size based on the size of improvement you care about. A small test set can catch a large regression, but it won't reliably settle a two-point advantage. If task outcomes are unstable, repeat runs and report uncertainty alongside the score.
The study's estimates - around 447 tasks to resolve a five-point difference under its observed conditions - aren't a universal formula. Use them as a warning against overconfidence, then estimate what you need for your own task mix and flip rate.
4. Diagnose mechanisms, not just scores
When one harness loses, dig into what went wrong. Check output caps, context limits, step limits, offline tool failures, retry behavior, and tool reliability before concluding the wrapper itself is inferior. In the study, output truncation explained a big portion of one OpenCode gap, but adding a continuation prompt didn't reliably fix it. A plausible mechanism isn't the same thing as proof that a particular patch will work.
A practical comparison checklist
- Keep model, tasks, and evaluation settings fixed while testing harnesses.
- Repeat the baseline configuration to estimate task-level flips.
- Log prompt and tool-schema tokens, steps, retries, and cost.
- Record failure reasons alongside pass/fail outcomes.
- Require the improvement to exceed normal run variation before migrating.
- Test the winning setup on representative production work before rollout.
The last step matters because a benchmark is a proxy. Passing repository tests is evidence - but deployment also involves review burden, latency, security boundaries, and the cost of a bad change.
Conclusion
A harness comparison is only as trustworthy as its baseline. In this study, reruns with the same configuration changed outcomes on 13% of tasks - the same median flip rate as switching harnesses. That makes small, one-off score gains hard to take at face value, especially on short task sets.
For engineering teams, the practical takeaway is straightforward: measure repeatability on your own workload, isolate model and harness changes, and track cost per solved task alongside pass rate. Then investigate the failure modes behind any score gap. Harnesses aren't a substitute for better models, and models aren't a substitute for reliable orchestration. The right choice is the setup whose verified benefits justify its operational cost.