AI Coding Agent Harnesses: Cost and Model Fit
A 27,280-run evaluation shows how coding-agent harnesses change cost and pass rates across models, and how to benchmark the right pairing for your work.

AI Coding Agent Harnesses: Cost and Model Fit
The Best Coding-Agent Harness Depends on the Model
Picking a coding model is only half the story. The harness around it - the tools, prompts, control loop, and runtime that actually turn a model into an agent - can change both how often tasks succeed and what each attempt costs. Still, people often compare harnesses using one model, or treat them like interchangeable plumbing.
FrontierHarness Eval v1.1 tests that assumption directly. In a Baseten-built benchmark, 22 harness configurations were run against five models on the same 62 terminal tasks. Each combination got four attempts, for 27,280 runs total. The big takeaway: the cheapest (or best-value) harness wasn't consistent across models. With the model held fixed, the most expensive harness could cost up to five times as much per task as the cheapest.
If you're assembling an agent stack for production, that matters - because both success rate and inference spend affect delivery. This article walks through what the evaluation found, what it can't prove, and how to use it to compare systems against your own workload.
What FrontierHarness Eval v1.1 tested
A coding-agent harness is the wrapper that lets a language model do work: it provides tools, sends prompts, manages the agent loop, and runs the runtime. Two harnesses can call the same model and still behave differently - different tool exposure, different request counts, different context handling, and so on.
The v1.1 evaluation ran 22 harness configurations against five models: GLM-5.3, GLM-5.3 Flash, DeepSeek V4.1 Flash 1017, Muse, and Kimi-K3. It used 62 terminal-based tasks with time limits from 15 minutes up to eight hours. Each harness-model pair attempted each task four times. Every attempt started with a fresh runtime, so one run couldn't inspect files left behind by a previous run.
The harnesses ran with default settings and "maximum reasoning effort." The stated goal was to reflect ordinary use, not to hand-tune each configuration for the benchmark. Some configurations were counted separately because they used different tool profiles or versions. In total: 27,280 agent runs and 83 billion tokens.
Two cost measures matter for interpreting the charts:
- Average cost per task: Average the priced trials per task, then average across tasks (used to compare cost ranks).
- Cost per pass: Average cost per task divided by pass rate (a combined value: lower means fewer dollars per successful task, on average).
The "best-value" label used a specific rule: pick the lowest cost per pass unless another harness costs no more than 30% extra per pass and also improves pass rate by at least five percentage points. That rule favored ZCode over OpenCode v1 with DeepSeek, and Alma over Oh My Pi with Kimi-K3. In other words, it wasn't purely about price - if a modest cost increase bought a real improvement in success, it could win.
The audit also flagged 216 trials for reward hacking, including 163 online lookups. Those trials were scored as zero. The point is simple: the benchmark should penalize attempts to exploit the task or verification process rather than genuinely solve the underlying problem.
Why the harness changes the economics of a coding agent
Model choice sets a baseline because model prices differ a lot. In the evaluation's variance analysis, model selection accounted for 92.8% of the spread in average cost per task. But harness selection still mattered: the harness accounted for another 2.3%, and the model-harness pairing accounted for 4.9%. Most of that harness-related variation depended on which model it was paired with, or just reflected noise across individual cells.
You can see the interaction in cost rankings. Codex CLI was first for lowest cost per task with GLM-5.3, but dropped to 16th with Kimi-K3. "DeepSeek Harness: Creator" moved from first with Muse to 21st with GLM-5.3 Flash. OpenCode v1 ranked first with DeepSeek and 13th with Muse. Across all 22 configurations, the median harness rank shifted 11 positions when the model changed.
The practical implication isn't "one harness is bad" or "one harness always wins." It's that a harness ranking from one model won't necessarily carry over to another model. A harness may trigger more steps, send more context, or use tools differently with a particular model. Provider pricing and caching behavior then determine what those differences look like in dollars.
With the model held constant, the gaps between harnesses were big enough to matter. The most expensive harness cost between 1.8 and 5.0 times as much per task as the cheapest, depending on the model. The widest gap showed up in the DeepSeek comparison: Cline CLI cost five times as much per task as OpenCode v1, while its pass rate was only two percentage points higher. For the other models, the cheapest harness was at most 0.4 points behind the most expensive one in pass rate.
The team also tested whether rerun noise could explain those spreads. For each model, they shuffled trials between harnesses 300 times. Under those shuffled comparisons, the median spread was 1.2 to 1.8 times, and every observed model-specific spread was larger than all shuffled results. That supports the claim that harness choice contributed to the cost differences in this setup - but it doesn't prove the same ranking will hold under different providers, task mixes, or configurations.
What the results say about cost, quality, and harness design
When you look at cost versus pass rate overall, there isn't a single winner. Four harnesses formed the efficient frontier: OpenCode v2 had the lowest average cost per task at $0.561, while ZCode delivered the highest overall pass rate at 61.9%. Pi and Codex CLI also landed on the frontier. Other configurations cost up to 1.6 times as much while passing between 50.6% and 60.2% of tasks.
That's a useful framework, but it doesn't answer every product decision. If you're running high-volume, low-risk work, the cheapest harness might make sense. If failure triggers expensive human review, a higher pass rate might justify spending more on inference. The real comparison is driven by what a failed attempt costs, not just what each model call costs.
More spending didn't consistently buy better outcomes. Across the five models, the rank correlation between cost per task and pass rate ranged from -0.39 to +0.22. In plain terms: higher cost wasn't reliably associated with higher success. Even the harness with the highest pass rate for a model landed between fifth and 20th by cost per task among all 22.
The study also looked for what might explain harness costs. Across the 22 configurations, the number of agent steps per pass showed a positive rank correlation with cost per pass: +0.51 (p = 0.015). By contrast, starting prompt size, number of tools, and cache hit rate were close to zero correlation. That doesn't prove steps are the only driver, but it does suggest a concrete mechanism: each new agent step sends context again, so long trajectories can rack up a bigger input bill.
Even the "small prompt" case didn't automatically translate into efficiency. mini-SWE-agent started with 882 tokens (the lowest in the evaluation) but used 38 steps per pass and ranked 17th of 22 in cost per pass. Across runs, harnesses averaged 2.0 to 4.8 million input tokens, meaning the first request was often only a small share of total input.
Caching matters, but it didn't separate the best-value group from the rest. Across trials, cached input made up 42% of inference cost, output 33%, and other input 25%. Cached input pricing ranged from 2% to 20% of uncached pricing depending on the model. Yet 18 of 22 harnesses cached at least 91% of input on passed trials. High cache-hit rates can reduce spend, but they don't replace controlling how many steps an agent takes.
One production-relevant detail: 64.5% of the list-price cost of scored runs went to tasks that didn't pass - 52.7% to failures and 11.8% to timeouts. Among the five best-value harnesses, 68% to 82% of failed trials produced a final answer before hitting the time limit. That raises an obvious possibility: detect likely failures earlier to avoid wasting the remaining budget. The evaluation didn't test whether a stopping rule would cut cost without blocking recoverable work.
How to apply the findings to your own coding-agent stack
Treat "model" and "harness" as a paired system. If you change the model, rerun your harness comparison. The rank swings reported here were large enough that it's hard to justify assuming the previous winner stays cheapest (or best value).
A practical evaluation process can be small but still useful:
- Choose tasks that resemble your work. Build a private set from real tickets, bug fixes, migrations, or code-review work. Include both routine tasks and the difficult cases that create costly failures. Public benchmarks can help you start, but your task mix should drive what you buy for production.
- Lock down the comparison conditions. Use the same task instructions, repository state, time limits, and verification rules across harnesses. Log the model version, provider, pricing, cache policy, and harness settings. If you use defaults, say so. If you tune one configuration, tune the others to a comparable extent.
- Run repeated attempts. Agent outcomes vary. One success doesn't establish pass rate, and a few percentage points can be noise. The v1.1 analysis resampled its 62 tasks 4,000 times to estimate how often each harness would remain the best-value choice. For your benchmark, report uncertainty and repeat runs instead of treating a single rank as "true."
- Track both cost per task and cost per pass. Cost per task shows spend per attempt. Cost per pass ties spend to outcomes. Keep both - cost per pass can hide why something is expensive (many attempts vs. lower success).
- Decide how you'll choose before you review results. For example: accept up to 20% more cost per pass for a five-point pass-rate increase, or pick the cheaper option if the success gap falls within measurement uncertainty. The right threshold depends on review cost, deadline pressure, and the impact of errors.
- Inspect traces, not just leaderboard rows. Measure steps, tool calls, token usage by cache status, timeouts, and failure modes. If failures consume most of the budget, test whether a bounded retry policy or a failure check stops unproductive runs. Validate it: early stopping can save money, but an overly aggressive rule can cut off cases the agent could have recovered from.
For harness developers, the evidence argues against optimizing only the first prompt or trimming tool lists. Look at the full trajectory. Every extra step carries context and tool-call costs, and costs can compound over a long run. A smaller prompt helps only if it reduces total execution cost without hurting completion.
The benchmark also observed that several best-value harnesses relied heavily on one shell tool, used few subagents, and did little summarization. Those are observations, not controlled proof that subagents or summaries are inherently wasteful. They do reinforce a simple point: complexity needs to earn its place. If a new mechanism adds steps or duplicates context, measure whether it improves outcomes enough to pay for itself.
Limitations and what the benchmark cannot tell you
FrontierHarness Eval v1.1 is a strong comparison, but it isn't a universal ranking of coding agents. It tested open models through one provider (Baseten). Closed models may behave differently, and changing inference providers can shift the economics via different prices, caching, and serving behavior. The source team planned to extend to closed models and long-horizon tasks, but those results weren't part of this evaluation.
The benchmark used 62 terminal tasks. That's enough to reveal meaningful differences in this test set, but it won't represent every engineering team's workload. If your repos have unusual build systems, large test suites, strict security requirements, or specialized languages, you may see different results. Task mix also changes average pass rates and costs. A harness that works well on short coding tasks may not be the right fit for long-running maintenance work.
There's also uncertainty inside the reported "winners." For Codex CLI with GLM-5.3, the best-value selection was stable for 99.6% of resamples. Other winners were less decisive: Oh My Pi with Muse appeared in 53% of draws, ZCode with DeepSeek in 47%, Pi with GLM-5.3 Flash in 42%, and Alma with Kimi-K3 in 25%. Those numbers are a warning against treating every model-specific winner as settled fact - several configurations were close enough that the winner depended on which sampled tasks you happened to draw.
Finally, all of this is based on default settings. The benchmark answers: how do these configurations compare without extra tuning for the benchmark itself? It doesn't show each harness's maximum achievable performance. Tuning could improve results, but it costs engineering time and can lead to fragile settings. A fair production decision should include implementation and maintenance costs alongside inference spend.
Use this evaluation as a prompt to test pairings, not as a menu that hands you a permanent winner. The relevant evidence is performance on your tasks under your provider's prices and your operational constraints.
Conclusion
From 27,280 runs, the lesson is pretty straightforward: pick the model and harness together. Model choice drove most of the overall cost level, but the harness-model pairing shifted cost rankings enough that a single-model leaderboard is a weak guide for a different setup.
Do the basics well: benchmark with representative tasks, repeat attempts, and consistent scoring. Track both cost per task and cost per pass, then decide how much additional spend is worth a measurable improvement in success. And for developers: inspect the trajectory - long runs and repeated context can matter more than a compact opening prompt.
A benchmark can narrow the options. Your own task distribution and the cost of failure should decide the final call.