Third-Party AI Audits: Anthropic METR Sets New Frontier Safety
Discover how Anthropic METR gives third-party AI audits employee-like access, boosting independent AI evaluation for frontier AI safety and AI regulation—what’s next?
Anthropic's AI Audit Experiment: What Real Independence Requires
AI companies are increasingly expected to explain how they test their most capable models. But "explaining" a safety process and letting an outside group examine it are not the same thing. An outside evaluator can only assess risks it can actually see - and labs have real reasons to keep some material private, from customer data to details that are commercially sensitive.
Anthropic is trying to address that tension. In an announcement covered by ZDNET, the company said six evaluators from the nonprofit METR would get what it called "employee-like access." That includes company hardware, office space, and internal tools and systems broadly comparable to what Anthropic's own risk teams use. METR is also expected to publish its findings - something Anthropic says it can't block - while being asked to redact sensitive information where necessary.
This could make outside evaluation more credible. But it doesn't automatically make evaluation mandatory, guarantee real independence, or settle who should oversee frontier AI. The more useful question isn't whether this is a breakthrough or a public-relations move - it's what evaluators can verify, what they can't, and what businesses and policymakers should expect from a third-party AI audit that's actually designed to hold up under scrutiny.
What Anthropic is offering METR
Anthropic said six METR evaluators would work with access to its facilities and internal systems. The company described that access as roughly comparable to what its own risk assessment teams can use. Employees were encouraged to share relevant information with the evaluators, the way they would with colleagues working on alignment risk.
There are clear limits. Anthropic identified categories outside the arrangement, including customer data, attorney-client privileged information, and details about model architectures. The company also said METR would publish its findings, while Anthropic could not suppress them, and that METR would redact sensitive material if needed.
That mix is important. This is more involved than handing an outside group a model endpoint and asking it to run a fixed set of prompts. It gives evaluators a chance to see some of the internal evidence behind the company's safety work. Still, it isn't unrestricted access to every system, record, or decision.
What "employee-like access" does - and does not - mean
The phrase describes a level of access, not a change in employment or governance. METR remains an external nonprofit evaluator. The arrangement doesn't mean METR staff become Anthropic employees, the company hands over control of its systems, or METR gains any special legal right to inspect everything it wants.
It helps to separate access from independence. More access can make an evaluation more informed, but independence depends on other factors too: who chooses the questions, who runs the process day-to-day, what can be published, and whether evaluators can disagree with the company without jeopardizing future access or funding.
The published findings part matters because Anthropic says it can't withhold them. Still, the redaction process is where a lot can go right or wrong. Redactions can be reasonable when they protect personal data, security-sensitive details, or genuinely confidential information. But redactions can also make a report harder to judge if readers can't tell what was removed or why. A serious setup needs clear rules about protecting sensitive specifics versus hiding conclusions.
For the six evaluators, the key practical test is whether the access provides enough context to check risks that may not show up in a public demo. Model behavior can shift based on how it's trained, deployed, monitored, and updated. Reviewing a model in isolation can miss important parts of the overall system.
Why access matters more than a published safety claim
Companies often publish safety policies, model cards, and evaluation summaries. Those documents can help users understand intended uses and known limitations, but they're still produced by the organizations being assessed. A third party can add value by examining evidence directly instead of relying only on the organization's own description of its controls.
That said, not every outside evaluation is equally useful. A narrow evaluation can answer a narrow question precisely while leaving other risks untouched. A model can look strong on a benchmark and still act unpredictably in a real product workflow - especially when it's connected to tools, or when users find ways around safeguards. The evaluation has to match the system and how people actually use it.
The gap between a lab's view and an evaluator's view
Anthropic said its goal was to narrow the gap between what outside evaluators understand about a frontier AI company's systems and what the company itself understands. That's a sensible aim: a lab might have internal evidence it can't safely or responsibly publish in full, while external reviewers might not have enough information to test the company's claims.
The difficult part is deciding what evidence is "enough." Depending on the question, evaluators might need more than output behavior. They may need access to testing methods, risk assessments, deployment safeguards, or incident-handling processes. More access doesn't automatically mean more certainty, but it can reveal assumptions that a public summary leaves unexplained.
For example, a business evaluating a model for customer support won't learn everything it needs from a headline benchmark. The real questions are whether the model reliably escalates high-risk cases, how it behaves when it's connected to customer records, and what happens when a user feeds it misleading information. A useful assessment ties model behavior back to the intended product, its permissions, and the company's operating procedures.
This is why third-party review is better seen as part of an ongoing feedback loop, not a one-time stamp of approval:
- Define the intended use and the harms that matter in that context.
- Test the model and the surrounding product controls against those risks.
- Document failures, limitations, and open questions.
- Fix the system or restrict use when the evidence supports it.
- Reassess after meaningful changes to the model, product, or deployment.
An audit that ends with a report but offers no route to remediation can satisfy the public-facing part without improving safety. A private evaluation that leads to changes but provides no accountability can leave customers and regulators unable to judge whether the process is credible. The best designs try to do both: real operational follow-through and enough transparent reporting to test the strength of the conclusions.
Why self-regulation has limits
AI labs have direct access to their models and development processes, so internal testing is necessary. But it isn't enough as the only source of assurance. The company that deploys the system also has commercial incentives, release pressures, and its own view of what counts as acceptable risk. That doesn't prove the company's safety finding is wrong. It does mean independent scrutiny has a role.
At the time of the reported announcement, US federal AI policy was described as relying largely on voluntary commitments rather than binding requirements for external model evaluation. The same coverage also pointed to proposals that would require independent verification for some companies and noted regulatory attention following safety and security incidents. Those developments raise the wider policy question: should independent assessment depend on a lab's willingness to share, or should there be defined obligations for systems that meet specified thresholds?
What this changes for AI regulation and business
Anthropic's arrangement is a company-led initiative, not a general legal requirement. That matters because it limits what other organizations should infer from it. It shows one possible model for giving an outside evaluator deeper access while still maintaining confidentiality boundaries. It doesn't establish a standard that everyone must follow, or guarantee that every qualified evaluator would get the same rights.
For policymakers, the key design problem isn't only whether to require audits - it's how to define evaluation in a way that produces real scrutiny instead of turning compliance into paperwork. A rule that asks companies to commission a report without specifying evaluator independence, access, publication, or follow-up may generate activity without reliable assurance. A rule that demands unrestricted disclosure could force exposure of personal information, security-sensitive methods, or legitimate trade secrets.
A workable regime would need to define review scope, set credible qualification standards and conflict-of-interest rules, and spell out what can be redacted. It would also need an answer for what happens when an evaluation finds a serious problem. The aim isn't to publish every technical detail. It's to make sure confidentiality doesn't erase accountability.
Implications for companies adopting AI
Businesses that buy or integrate AI shouldn't treat a headline audit as a substitute for their own due diligence. A model-level evaluation can answer some important questions, but it can't fully cover a buyer's data flows, user permissions, escalation processes, or regulatory obligations. Those depend on the product and the operating environment.
Procurement teams can make an external evaluation more useful with a straightforward set of questions:
- What was evaluated? Identify the specific model version, product configuration, and deployment context. A report about a different version may not match what you're actually buying.
- Who conducted the work? Look for relevant technical capability, independence, and disclosed conflicts. A third-party label alone isn't enough.
- What could the evaluator inspect? Clarify whether the review stayed within public documentation and outputs or included access to internal evidence.
- What were the boundaries? Ask what was excluded, how redactions were handled, and whether those limits affect the conclusions.
- What happens after a finding? Determine whether issues led to mitigations, restrictions, retesting, or a documented acceptance of risk.
- What must your organization still test? Assess the model in your own workflow, using your own data controls, users, and failure-handling process.
These questions apply whether you're a US company navigating emerging federal and state requirements or a UK business working under privacy and governance obligations such as the GDPR and UK data protection law. The details vary by jurisdiction and use case, but the operational principle is steady: assurance about a model doesn't automatically carry over to every product built on it.
Conclusion: treat access as a starting point
Anthropic's decision to give METR evaluators deeper access matters because independent review only helps when reviewers can examine evidence that's actually relevant to the risks being assessed. The arrangement also makes a point that's easy to lose in headlines: access, independence, and transparency are separate issues. A reviewer can have more access and still not get every record, and a published report can still be hard to judge if its scope and redactions aren't clear.
For AI companies, the practical challenge is to make outside evaluation rigorous without exposing information that should remain protected. For regulators, it's to define obligations that create genuine scrutiny rather than a compliance ritual. For businesses adopting AI, an external report is one input - not a replacement for testing the system in your own context.
When you're assessing any AI audit, start with five questions: What was tested? What could the evaluator inspect? Could the evaluator publish an unwelcome finding? What was excluded? And what changed as a result? If those answers are clear, the audit can support a real decision. If they aren't, "independent" by itself isn't enough.
Read ZDNET's report on Anthropic's METR evaluation arrangement and review METR's work on AI model evaluation before using this case as a benchmark for your own AI governance or procurement process.