CWE-bench v2: A Practical Guide to AI Security Testing
Explore how CWE-bench v2 tests AI agents on real code vulnerabilities, what its results reveal about capability and cost, and how to run the benchmark.
CWE-bench v2: What a Real-World Security Benchmark Reveals About AI Agents
AI coding agents can generate a patch that looks convincing without actually fixing the underlying vulnerability. They can also "fix" the reported issue while breaking something users rely on. That's why a benchmark has to test more than whether a model spots a familiar code pattern.
CWE-bench v2 takes a more grounded approach: it gives an agent a real open-source repository, a short instruction, and a specific security weakness to address. Then it checks the result with deterministic tests. The benchmark's public release includes 100 tasks across 91 Common Weakness Enumerations (CWEs), so it exercises agents across different vulnerability types while requiring them to keep existing behavior intact.
This article walks through what the benchmark measures, how its quality controls and scoring work, what the reported results suggest about capability and cost, and how teams can run the public tasks. It also covers what the benchmark can - and cannot - tell you when you're choosing or building an AI coding agent.
What does CWE-bench v2 test?
CWE-bench v2 evaluates whether an AI agent can find and fix a security weakness in a real software repository without breaking the software's existing functionality. The agent gets a repository checkout and a short task instruction. It must produce a change that satisfies the benchmark's verifier.
That setup matters. Test a model on isolated code snippets and it may recognize a "textbook" flaw - then fail when the real fix depends on the surrounding behavior. Real codebases have dependencies, interfaces, and assumptions that aren't always visible from a single function. A security patch also has to fit that context.
From vulnerability recognition to useful repair
The benchmark includes 100 tasks spanning 91 CWEs, a taxonomy of common software weaknesses. The intent isn't just to check whether a model can reproduce a known pattern. Agents have to interpret the instruction, inspect the repository, make a change, and satisfy checks for both the security behavior and the project's existing functionality.
In practice, that looks more like the job of a defensive coding agent than a quiz that asks for a vulnerability label. A patch that "seems reasonable" isn't enough - the code has to behave correctly for the conditions defined by each task.
The benchmark also tries to probe system limits. A defender can't pick and choose only the weaknesses an agent has seen before; attackers try to exploit whatever surfaces remain. Broad coverage is useful, even though 91 CWE categories aren't the whole of software security. A benchmark can sample a lot of ground, but it can't cover everything.
Public tasks and held-out evaluation
The public set gives researchers and engineering teams a shared set of tasks to run. CWE-bench v2 also maintains a held-out set, meant to track progress on tasks models may not have encountered. Evaluation on the held-out set is handled separately by the benchmark team.
This split addresses a common evaluation problem: once tasks are public, models may see them during training or tuning. Then a high score can reflect familiarity, not just general capability. Held-out tasks help test whether improvements carry over to unseen cases.
Still, no single score fully captures security capability. The benchmark is more like one piece of an evaluation system - especially if teams report which set they used, the model version, the number of attempts, and the execution conditions.
How does the benchmark measure quality and fairness?
A benchmark is only useful if its tasks and graders are solid. If the instructions are oddly artificial, verifiers reward only one narrow implementation, or starter code already passes, the score starts measuring test design rather than model capability.
CWE-bench v2 describes a review process meant to catch those issues. Each task is checked by hand, and the programmatic checks are curated with help from panels of language models. Those panels help shape the checks to be harder and fairer, while the final scoring remains programmatic and deterministic.
What the task review checks
For each task, reviewers look for things like:
- The instruction reads like a real request, not an exam prompt.
- The expected behavior can be inferred from the instruction.
- The verifier checks behavior, not one specific code structure.
- The starter code fails the relevant checks and the reference fix passes them.
- The fix preserves the original functionality.
- The verifier rejects memorized solutions, including known upstream patches.
These checks target different failure modes. If an instruction is vague, a correct solution can become indistinguishable from guessing. If a verifier expects an exact code structure, it can reject valid alternatives. And if a known patch passes without meaningful reasoning, results can inflate.
A key distinction here is between deterministic scoring and automated task design. Model panels contribute to shaping the checks, but the reward comes from programmatic verification. That matters for repeatability: two teams running the same task and conditions should not depend on a grader's subjective judgment.
What deterministic scoring does not guarantee
Deterministic doesn't mean comprehensive. A verifier can only check what its tests cover. A patch might pass those checks and still contain an issue the benchmark doesn't test. And the reference fix is a quality signal, not proof that every acceptable solution has been captured.
That's why human task review still matters - and why public feedback can improve the dataset over time. When tasks are open, researchers and developers can inspect them, reproduce results, and point out cases where an instruction or verifier is misleading. Transparency makes criticism possible, even if it can't remove the need for it.
If you're using scores to make decisions, the right takeaway is narrow: the benchmark measures performance on a reviewed set of security-repair tasks under specified conditions. It does not certify an agent as safe to deploy against production repositories without review.
What do the results say about AI security agents?
In the reported evaluation of 11 models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4. Claude Opus 5.5 follows with 67.3% pass@1 and 78.0% pass@4. At least one evaluated model solves 95 of the 100 public tasks, which suggests that most tasks are solvable by today's systems under the benchmark setup.
The remaining gap is operationally meaningful. Even the top model fails 19% of tasks across four attempts, based on its 81.0% pass@4 result. That matters if you expect an agent to review or repair code with limited supervision. It also shows that strong average performance and dependable task completion are not the same thing.
Read pass@1 and pass@4 together
Pass@1 is the share of tasks passed on the first attempt. Pass@4 measures performance across four attempts per task. The second number can show whether extra attempts help, but it shouldn't be treated as the same as having a solution that works reliably in one run.
For a product team, retries cost money and time - and they can add human review work. An agent that eventually solves a task after several tries can be useful in workflows that tolerate retries. It may be a bad fit for an interactive fix where you need a dependable answer quickly.
So the results point to two different questions:
- Can the agent solve the task at all? Pass@4 gives one answer.
- How often does it succeed in a single run? Pass@1 is usually more relevant for low-latency workflows.
Neither metric tells you whether a successful patch is easy to review, appropriate for your codebase, or safe to merge without human sign-off. Those are workflow and risk questions, not properties that a pass rate can settle by itself.
Capability has a cost curve
The reported cost comparison places DeepSeek V4.1 Flash (1017), Claude Opus 5.5, and Gemini 4 Argon on the pass@1 cost-performance Pareto frontier. DeepSeek hits 57.0% pass@1 at an average of $0.04 per rollout. Claude Opus 5.5 adds about ten percentage points at $1.83, while Gemini 4 Argon comes in at $3.92.
A Pareto frontier is helpful because it shows options that aren't dominated on both score and cost. It doesn't identify a single universal winner. A high-volume internal triage tool might care most about cost per rollout, while a security-sensitive workflow might accept higher costs for a better chance of a first-pass solution.
Treat these numbers as a snapshot, not a guaranteed quote. Pricing, model versions, prompts, and runtime choices change. Before adopting an agent, rerun the evaluation under your own conditions and account for the total cost of failed runs, retries, and human review - not just the model calls.
How can you run CWE-bench v2?
CWE-bench v2 tasks are packaged as self-contained Harbor tasks. Each one includes a Dockerized environment, an instruction, and a deterministic verifier. The dataset also includes a task registry, which lets Harbor fetch and run the benchmark from its hosted dataset rather than requiring you to clone and assemble everything manually.
Access to the dataset is gated. Accept the access conditions, then set up your Hugging Face credentials so Git can retrieve the task repository. The benchmark uses Harbor to run four attempts per task. Across 100 tasks, that comes to 400 trials, and the reported score is calculated as the mean reward over those trials.
A practical run checklist
Before comparing your results to a published score, confirm:
- You've accepted the dataset's access conditions.
- You've authenticated with Hugging Face and saved the token to Git's credential helper, as required for retrieval.
- Harbor and the agent you want to evaluate are installed and configured.
- You selected the intended model and allowed the environment to reach its model API host where needed.
- You set the attempt count to four if you want to match the pass@4 run.
- You recorded the model version, agent settings, runtime, and cost along with the score.
The launch command uses Harbor's repository option, the dataset identifier and version, the attempt count, agent and model settings, and Docker as the environment. Use the current dataset instructions for the exact command and repository identifier - copied commands may not match if credentials or model settings differ.
Evaluation and training are related, but not identical
The tasks are tagged as reinforcement-learning environments. That means the same environments and verifiers can provide reward signals for training defensive agents and for evaluating them.
That's useful for experimentation, but it raises a measurement concern: training directly on public benchmark tasks can make later evaluation on the same tasks less informative. If you train on the public set, keep a separate held-out evaluation and document how the agent was trained. A score only means what the model may have already seen.
What should teams take away from the benchmark?
CWE-bench v2 is best understood as a reproducible stress test - not a guarantee of security. Because the public tasks are shared, you can compare agents on the same kind of repository-based repair work. The broader CWE coverage also tests more than one familiar vulnerability template. Deterministic checks support repeatable scoring, and the cost figures make trade-offs visible that raw accuracy can hide.
For a team evaluating an agent, use the benchmark to narrow questions, not to skip local validation. A sensible evaluation process looks like this:
- Start with task fit. Check whether your real work resembles repository-level vulnerability repair, not just general coding.
- Compare first-attempt and retry performance. Decide whether retries are acceptable in your workflow.
- Calculate the complete cost. Include inference, execution, retries, and developer review.
- Inspect failures. Group them by weakness type and failure mode; an aggregate score won't tell you where the agent is brittle.
- Test on your own code. Dependencies, conventions, and threat models differ across benchmark repositories and production systems.
- Keep a human approval boundary. Passing tests can support review, but it shouldn't be treated as automatic authorization to merge security-sensitive changes.
For product builders, the results also clarify where automation may fit today. An agent can be useful for proposing patches, exploring a backlog item, or helping a security engineer investigate a report. The benchmark doesn't establish that unattended remediation is safe. Whether it's safe depends on the blast radius of a bad change, how well your tests capture intended behavior, and how quickly a human can spot a failure.
Making the open set available makes these discussions more concrete. Teams can reproduce results, compare models, and add evidence about tasks that need revision. In practice, that transparency is more valuable than a single headline score, because it gives evaluations a way to improve as systems and real-world requirements evolve.
Conclusion
CWE-bench v2 provides a practical way to test AI agents on security repair in real repositories, with broad vulnerability coverage, reviewed tasks, and repeatable programmatic scoring. The results show that leading models can solve many tasks - but there's also a long tail left to address, and stronger performance can come with higher cost.
Use the benchmark to compare systems and identify failure patterns, then validate those findings on your own code and workflow. For security-sensitive changes, an agent's score is useful evidence - not a replacement for engineering judgment or review.