Claude Haiku vs Opus: AI Bug-Fix Benchmark 🔥 (2026)

25 bugs, three models, three attempts per bug

The benchmark used 25 bugs that were fixed upstream between August and October 2026 across 13 open-source projects. The projects included JavaScript and Python libraries such as js-yaml, marked, markdown-it, zod, tomlkit, cachetools, and Python-Markdown.

Each model attempted every bug three times, for 225 total runs. The bugs were split into 15 "easy" cases (symptom and cause were relatively close) and 10 "hard" cases (the reported symptom could be far from the underlying cause).

To reduce the chance that a model would effectively "remember" the upstream patch, each run started from the commit just before the upstream change. Dependencies were installed, but the run started with no Git history. The prompt described the observed behavior, how to reproduce it, and what the expected result should be. It intentionally excluded the original cause analysis and the proposed fix.

The models ran through Claude Code 2.1.295 in headless mode, using medium effort. They could read and edit project files and run the project's tests, but they couldn't install new packages or access the network. So these numbers reflect model-assisted work in this environment, not every IDE, agent setup, or coding workflow.

What counted as a clean fix

A "clean fix" meant two things: it had to pass the maintainers' tests for the bug, and it had to avoid breaking other tests in the project. Tests written by the agent during the run weren't used to score success. That choice helps stop a model from getting credit just by authoring tests that align with its own patch.

It's a solid yardstick, but it's not the same as "approved for production." Passing tests doesn't guarantee the patch is maintainable, correctly scoped, secure, or consistent with the project's design. Human review is still the gate.

The benchmark's author also published the 225-run dataset, including cases passed, tests broken, time, and cost. The underlying write-up is on DEV Community. Those links are worth checking if you want to see individual outcomes instead of only the averages.

The average hides where the models differed

For 22 of the 25 bugs, Haiku and Opus ended up with the same number of clean fixes. The differences clustered in three cases:

• On a marked bug involving a link inside link text, Opus fixed it cleanly in 2 of 3 runs; Haiku didn't in any. All three Haiku runs failed the spec test for that example.
• On a markdown-it bug involving an unclosed link label hiding a code span, Opus passed all three runs. Haiku passed one; two runs fixed the reported issue but broke nearby tests.
• On a js-yaml whitespace-only block scalar bug, Haiku passed all three runs. Opus had one clean fix, and Sonnet had none.

There was also a zod case where the report involved numeric record keys needing to be handled in multiple parts of the JSON Schema output. No model completed the whole fix. Across the 24 test cases for that bug, Opus passed 14, Haiku passed 9, and Sonnet passed 5. That's the kind of task where a simple "solved vs not solved" can miss what's really happening - partial progress can vary a lot.

The cost gap can also be easy to misunderstand. Haiku's $0.019 per run is about 95% below Opus's $0.38 in this setup. That matters if you run lots of similar tasks. But it isn't, by itself, proof that Haiku is the best overall choice. Engineering review time, retries, debugging effort, and the cost of a bad patch all figure in.

Opus was more reliable on the hard round

In the hard round specifically, Opus had 16 clean fixes, Sonnet had 14, and Haiku had 12. The bigger difference is regressions: Haiku broke other tests in 10 of its 30 hard-round runs, while Opus did so in 5.

That doesn't prove a stable ranking - this is still a small sample. But it does line up with something teams feel in practice: when the change touches multiple behaviors, the "looks fixed" patch can come with hidden costs. A less capable model may resolve the reported symptom while introducing regressions elsewhere.

So the practical takeaway isn't "cheap model vs smart model." It's the expected cost of the full workflow. For bounded, easy-to-validate tasks, the lower per-run cost can dominate. For changes that cut across several parts of a codebase, a regression can outweigh the API savings.

The statistical result is inconclusive for Haiku versus Opus

Across all 25 bugs, Fisher's exact test on clean fixes returned p = 0.86. When resampling whole bugs, the 95% interval for Haiku minus Opus runs from -11 to +5 percentage points. In plain terms: the data don't provide evidence that Opus fixed more bugs overall. They also don't rule out a gap of about 10 percentage points.

On the hard round alone, the interval ranged from -33 to 0 points. With only 10 hard bugs, that result is especially hard to interpret. It's reasonable to treat Opus's lead there as a signal worth testing in your own codebase, not as settled truth.

Sonnet passed fewer test cases than Opus overall. The paired permutation test reported a statistically significant difference (p = 0.006). Even then, this only applies to this set of tasks and evaluation method. It doesn't mean Sonnet will consistently underperform on other workloads.

The author estimates that showing equivalence within plus or minus 10 percentage points would require several hundred runs per model on different bugs. The punchline: "we didn't detect a difference" is not the same as "we proved the models are equal."

Every model missed some of the same cases

Here's a limitation that model size didn't fix. Three hard bugs exposed a constraint that showed up for all three models. In every run, all three models addressed the example described in the report, but they missed a related case that the report didn't mention:

• For cachetools, the report showed one way to call a cached method through its class. Maintainership tests covered other decorator variants that the models didn't handle.
• For js-yaml, the report described a byte order mark at the start of a file. The tests also covered a byte order mark before each document in a multi-document stream. The models missed that variant.
• For Python-Markdown, the models handled the reported HTML comment behavior but broke 3-5 tests involving comments after other inline HTML.

This is probably the most useful lesson for teams using coding agents: a larger model can't reliably guess every requirement when the task context is incomplete. Tests, surrounding code, documentation, and maintainer expectations are part of what the "real" bug-fix task includes. If that context isn't present, you can get a plausible patch that's still incomplete.

In the hard round, between 47% and 60% of runs weren't clean fixes, depending on the model. That's where your review effort should concentrate. Passing the reported example isn't the finish line - reviewers still need to check nearby cases, regression risk, and assumptions the report leaves implicit.

Read the limitations before generalizing

A few details make the results harder to generalize:

• Each bug was tested only three times per model, across 25 bugs. The totals can move noticeably with a few changes.
• Unlike an earlier experiment mentioned by the author, every run here could run the project's tests. That means it's not directly comparable to that earlier result.
• The tomlkit report was written by the author from an upstream issue without a reproduction. All models passed the same 20 of 29 cases, so it's probably not a clean measure of the original report.
• On Windows, the zod project's CLAUDE.md symlink didn't extract, so those runs lacked project-specific instructions.
• Timing tests were excluded from the regression count because they fail unpredictably under load.
• The projects are public, so some similar code might have appeared in training. Still, the tested fixes were made between August and October 2026.

There's also a disclosure: the benchmark author runs TokenSave, which publishes the measurements, and reports receiving no support or credits from Anthropic. That doesn't automatically invalidate the results, but it's still worth reading the method and the dataset directly - just like you would with any benchmark.

Use a lower-cost model when the task is bounded

Haiku is a reasonable default when the bug has a clear reproduction, the affected behavior is localized, and the project has tests that can catch a bad patch quickly. In the easy round, Haiku had 40 clean fixes out of 45 - better than Opus's 38 - and it did it at a much lower per-run cost.

That doesn't mean every "easy" bug is safe to farm out. Small parsing or edge-case changes can still have broad compatibility implications. The real test is whether the change has a narrow blast radius, and whether a human can verify it cheaply.

Consider a stronger model when the change crosses boundaries

Opus makes more sense when the bug touches several interacting behaviors, the symptom is ambiguous, or the cost of a regression is high. In this benchmark, Opus led on the hard round and broke fewer existing tests.

That's not proof that Opus is universally superior, but it's a reasonable hypothesis to try on your own work. You can also use escalation instead of forcing everything through the top model: start with a lower-cost model, and only switch when tests fail, the patch spans multiple modules, or the issue requires reasoning across multiple representations.

Escalation only saves money if it's deliberate. If you just rerun the same prompt with a different model and add nothing else, you can end up repeating the same failure mode with a higher bill.

Make tests and review part of the model budget

Model selection isn't just about tokens or API prices. You need to track more than that:

• The share of patches accepted after human review
• How often tests catch regressions before review
• Retry and escalation rates
• Time spent reviewing, correcting, and integrating patches
• The severity and frequency of failures that reach later stages

A cheaper model can still cost more if it needs multiple retries or produces patches that require substantial correction. A higher-priced model can be a waste if it's solving a mechanical problem the cheaper one handles reliably. The metric that matters is cost per safely accepted change, not cost per API call.

Start with a representative sample of your own bugs. Keep prompts, repository state, model versions, test commands, and acceptance criteria consistent. Use more than a handful of issues, and record failures as carefully as successes. If you change the agent or the test environment midstream, note it - those details are part of what you're evaluating.