MMLU-Pro Answer Key Errors: What Model Consensus Finds
A MMLU-Pro audit tests whether leading AI models can spot wrong answer keys, what consensus catches, where it fails, and why human review still matters.
Can AI Model Consensus Find Wrong Answers in MMLU-Pro?
A benchmark answer key looks like a fixed point: a model is right or wrong, and the score follows. But that assumption breaks the moment the key itself is wrong. In a widely used test set like MMLU-Pro, a bad label can skew model comparisons - making a capable system look worse - and it can also reward models that simply repeat the mistake.
An audit by AI agents working in MV2's name at Munim, Inc. asked a practical question: if multiple strong models agree, can that help surface wrong labels? The result is promising, but it's not a magic "declare the benchmark broken" shortcut. At one pre-registered threshold, model consensus flagged expert-confirmed wrong keys far more often than it flagged questions experts had cleared. Still, most flags didn't come with an expert error label, and agreement alone can't tell you which answer is actually correct.
This article explains how the audit was run, what the results do and don't show, why MMLU-Pro's extra answer options complicate interpretation, and how benchmark teams can use model consensus as a review signal without treating it as ground truth.
How did the MMLU-Pro audit test model consensus?
MMLU-Pro includes 12,032 multiple-choice questions. Some come from MMLU, while others come from a STEM website source. The maintainers publish model responses for each question, so researchers can compare those responses against the benchmark key at scale.
The audit started from a specific problem. An earlier review looked at 46 MMLU-Pro questions whose keys MMLU-Redux 2.0 experts had marked wrong. In that earlier analysis, all 45 published model archives that were included picked the experts' answer more often than the benchmark key. But the expert labels covered only part of the MMLU-derived questions, and none of the questions from the other source. So the audit asked: could model agreement help find candidate errors outside that reviewed subset?
The rule was simple - and testable
The researchers chose the 10 most accurate of 45 usable model archives, ranging from Gemini 3.1 Pro (91.1% accuracy) to Gemini 1.5 Pro (70.3%). A question was flagged if:
- at least six of the 10 chose the same answer option,
- that option wasn't the key, and
- more models chose it than chose the keyed answer.
This rule matters because it reduces the temptation to treat disagreement from a single model as meaningful evidence. It also stays reproducible: the same model answer files and the same threshold should produce the same list of candidates. The audit reports that these models hadn't seen MMLU-Redux labels, so the comparison to expert judgments was meant to be independent rather than a re-run of what the experts already taught.
Before running the analysis, the team recorded six predictions. Four ended up supported and two didn't. That distinction matters: the study wasn't just hunting for examples that fit a preferred story - it also tracked where the hypothesis didn't match the data.
What counts as independent evidence?
Even with those precautions, the expert labels aren't creating 10 independent judgments. The archive includes runs that can overlap in model families, training data, or shared failure patterns. The audit notes that the set contains three Claude 3.5 Sonnet runs, two GPT-4o versions, and two runs labeled arx. Models like that can share correlated errors, meaning "agreement" can reflect common blind spots rather than independent reasoning.
So the signal isn't worthless - but it supports a narrower claim. Strong consensus can help prioritize questions for review. It can't prove that the majority answer is correct. That constraint shows up throughout the audit's findings.
What did the audit find about answer key errors?
The clearest result: the consensus rule flagged expert-labeled wrong keys much more often than it flagged questions experts had cleared.
- It flagged 36 of 55 expert-confirmed miskeys (65.5%).
- Among the 2,553 questions experts considered fine, it flagged 294 (11.5%).
That's a large gap. In other words, the rule flagged known miskeys about 5.7 times as often as expert-cleared questions. The audit also caught 34 of 46 inherited miskeys from the previous review. For 31 of those 34, the panel's preferred answer matched the experts' answer.
| Question group | Questions flagged | Flag rate |
|---|---|---|
| Experts say the key is wrong | 36 of 55 | 65.5% |
| Experts found another error | 29 of 117 | 24.8% |
| Experts say the question is fine | 294 of 2,553 | 11.5% |
| MMLU questions not reviewed by those experts | 501 of 3,928 | 12.75% |
| Questions from the STEM website source | 271 of 4,083 | 6.64% |
| All MMLU-Pro questions | 1,210 of 12,032 | 10.1% |
Why a useful signal still produces false leads
There's a catch. Among the 359 flagged questions that had expert review labels, 65 had some kind of expert error label - 36 wrong keys plus 29 other errors. That's 18.1%. Put another way: roughly four out of five flags in that reviewed subset came from questions experts had already cleared.
This is the trade-off the method makes. It has decent recall for known miskeys, but a review team still has to inspect lots of candidates that may not be errors. Some could reflect a shared model misconception. Others might have more than one defensible answer, or a reviewer might have missed something. The audit data alone can't distinguish which explanation applies to any specific flagged question.
The audit also shows what happens under a stricter threshold. When the rule required unanimity across all 10 models, it flagged 201 questions. It caught 21 of the 55 expert-confirmed miskeys. In that case, 36.4% of the flagged, expert-reviewed questions carried an expert error label. Requiring unanimous agreement improved the "hit rate" among reviewed flags - but it missed more known errors.
Neither threshold is automatically best. A team with more capacity for triage may prefer the six-of-10 rule. A team with limited review time might choose a stricter threshold, accepting lower coverage. The right choice depends on whether the priority is finding many potential issues or reducing the number of human reviews per confirmed issue.
The source of a question changes the result
The audit also tested a prediction about where questions came from. The MMLU-Pro paper identifies the STEM website source as a place where wrong keys had been found, so the researchers expected those questions to receive more consensus flags.
The data went the other way. Questions from that source were flagged at 6.64%, about half the rate for MMLU-derived questions (roughly 12.8%). This difference doesn't prove the STEM-source keys are cleaner. One plausible explanation is that numerical questions distribute errors across answer options, making it harder for models to converge on the same wrong choice. But the evidence in this audit doesn't settle which explanation is correct.
Do MMLU-Pro's added options explain the flags?
MMLU-Pro expands the original MMLU answer choices. It keeps the original four options and adds up to six more. The MMLU-Redux experts reviewed only the original four, not the expanded set. That means there's a realistic blind spot: if a correct answer appears only in the added options, the experts could judge the original key as wrong or right without seeing the full MMLU-Pro question.
To test that possibility, the follow-up audit mapped MMLU-derived questions back to MMLU, labeled each answer option as original or added, and looked at which type the model panel chose. Of the MMLU-derived questions, 5,818 retained all four original options, which made them usable for this analysis.
Added options account for part of the disagreement
Among flagged questions that experts had cleared, the panel selected an MMLU-Pro-added option 185 times out of 290 (63.8%). That's higher than the rate at which the panel's "scattered" wrong answers landed on added options elsewhere (56.7%).
This could suggest experts missed correct answers introduced during MMLU-Pro's expansion. But you can't stop at that observation - you need a denominator. Added options made up 64.1% of the non-key options in those questions. The panel picked added options at almost exactly the rate you'd expect from that proportion.
There's also a counterpoint. In 105 of the 290 flagged questions, the panel picked one of MMLU's original distractors. Those are options the experts had already seen and still cleared the question. So added options explain part of the disagreement - but not all of it. And a model preferring an added option doesn't automatically mean that option is correct.
The researchers expected the effect to be strongest in law questions. That prediction didn't hold up. The panel chose an added option in 69 of 139 flagged law questions, which lines up with 61.9% in the rest of the dataset. So, even when an analysis points in the right direction, it shouldn't be treated as confirmation by itself.
What benchmark teams can take from this
Expanding the answer set changes more than the difficulty of a question. It changes the review surface. If experts assess only the original options while models are being evaluated against the expanded set, disagreements can reflect a mismatch in what was reviewed - not just a straightforward key error.
So for benchmark operators, the practical step is to preserve option provenance and review history. Record which options were original, which were added, and which version experts saw. When a consensus flag points to an added option, send the question for human review using the full option set and source context. Don't automatically swap the key.
How should benchmark teams use model consensus?
The point isn't "models can audit benchmarks." It's narrower: "model agreement can help allocate human review." That framing is easier to defend - and it leads to workflows you can actually run and document.
A practical review workflow
- Freeze the benchmark version. Keep question text, answer options, key, and source metadata tied to a version. Otherwise, a later review may not reproduce the same disagreement.
- Define the flag rule before looking at results. Choose the panel, consensus threshold, and tie handling in advance. Pre-registration helps avoid tuning the threshold to make results look better after the fact.
- Track model independence. Record model families, versions, and repeated runs. Ten output files don't always mean ten independent systems - especially if they share the same model family.
- Separate candidate types. A key disagreement, a potentially ambiguous question, and a possible issue involving an added option are different review situations. Put them in different queues, instead of labeling every flag as "miskey."
- Ask humans to verify the underlying claim. Reviewers should check the question, all answer options, the cited or implied sources, and the intended interpretation. A majority vote isn't a substitute for subject-matter review.
- Record the disposition. Mark each candidate as confirmed key error, another question error, acceptable alternative, model error, or unresolved. Over time, this feedback helps estimate how precise the method is on your own benchmark.
This approach keeps the results auditable. It also lets a team measure whether consensus review actually saves time. If reviewers consistently dismiss flags from one model family or one question type, the panel or threshold can be adjusted in the next pre-registered run - not rewritten after the fact.
Use the signal where the incentives match
Consensus review makes the most sense when the cost of an unnoticed label error is high and the flagged questions can be checked by qualified reviewers. It can help prioritize large evaluation sets, especially when you already have per-question model outputs.
It's less useful as a fully automatic correction system. A benchmark used for research or procurement shapes published comparisons and purchasing decisions. Changing keys based on model majority introduces a new source of error - one with unclear governance. The final decision should stay tied to a documented human review.
The audit also surfaced four inherited miskeys where all 10 models picked the MMLU-Pro key instead of the experts' answer: question IDs 6639 (virology), 8972 (abstract algebra), 9677 (astronomy), and 9947 (conceptual physics). Three of these depend on interpreting MMLU-Redux corrections written as bare digits. Those look like good candidates for human checking rather than evidence that the experts were necessarily wrong.
Key takeaways
- Model consensus flagged known wrong keys far more often than expert-cleared questions, but it didn't prove that a flagged alternative was correct.
- At the six-of-10 threshold, only 18.1% of flagged, expert-reviewed questions carried an expert error label.
- Requiring unanimous agreement improved the share of reviewed flags with error labels, while catching fewer known miskeys.
- Added options explain some disagreements, but many flags pointed to original distractors experts had already reviewed.
- Treat consensus as triage. Human verification, version control, and transparent dispositions are still necessary.
Conclusion
This audit makes a solid case for model consensus as a benchmark-maintenance tool - and just as importantly, it warns against overclaiming. A pre-registered rule found known MMLU-Pro key errors at a substantially higher rate than it flagged expert-cleared questions. But most flagged, expert-reviewed questions still didn't come with an expert error label, and the model panel wasn't fully independent.
For teams maintaining evaluations, the decision is straightforward: use consensus to prioritize human review when it's worth the effort, but don't let model votes rewrite the benchmark. Preserve the exact version and option provenance, document the review outcome, and treat unresolved cases as unresolved. That's how a noisy signal can improve benchmark quality without turning into another unexamined source of labels.