LLM Judges for Identity Resolution at Lower Cost
See how confidence-based routing pairs a fast decision model with a frontier LLM to audit identity resolution more accurately while controlling cost.
How to Audit Identity Resolution Without Paying Frontier-Model Prices
Identity resolution decides which data records belong to the same real person. If it gets that wrong, a customer profile can merge two people - or split one person across several profiles. Either way, analytics and personalization take a hit. Over-merging is particularly nasty: once records get joined, there's often no clean "answer key" to verify against.
You can review profiles by hand, but it's slow, and a small sample can miss rare mistakes. LLMs can speed up inspection at scale. The catch is simple: using a strong model for every decision can get expensive, but a cheaper model can be confidently wrong on the hard cases.
A benchmark comparing decision models Jev and GPT-6 Luna with frontier models found a practical middle path: run a fast model first, then send uncertain cases to a stronger model. On the easier benchmark, a Jev-plus-Opus setup hit 98.8% accuracy at $0.88 per 1,000 identities (versus $3.86 for Opus alone). That isn't a universal pricing promise. It's a sign that confidence calibration can make model routing work - if you test on cases that look like your data.
Why identity resolution is difficult to audit
Identity resolution (IDR) maps records from different systems to real people. A retailer might have a loyalty account, online transactions, and store-pickup records for one customer. IDR tries to connect those records to a single identity so teams can view the customer's activity as one story.
It sounds like a straightforward matching problem, but the data is rarely tidy. Names come misspelled or abbreviated. Addresses change. A field might be blank, copied from a relative, or filled with a placeholder. Dates get mistyped - or have month and day swapped. Some differences are harmless noise; others point to different people.
Why over-merging carries particular risk
An over-merge happens when one identity contains records from more than one real person. For example, two family members can share an address, or a parent and child can have similar names. When those records get combined, downstream systems can attribute purchases, support history, or communications to the wrong person.
Under-merging is the opposite: one person shows up as multiple identities. That can understate purchase frequency or lifetime value. Both errors matter, but over-merging also raises direct privacy and user-experience problems - like showing someone's information that belongs to a different person.
Why sampling needs more than a quick spot check
For a live customer graph, you usually don't have a complete ground-truth answer key. If every record already had a verified person identifier, you'd be mostly done. Instead, teams rely on signals - like "this profile has an unusually high number of emails." Those signals can flag problems, but they don't prove the profile is wrong. Human review helps, but it's still limited by time, and sampling can miss rare errors.
The benchmark's authors estimate reviewers can examine about 50 identities per hour. That makes large audits expensive in staffing time. And if over-merges are uncommon, a small sample can give you a misleading error-rate estimate. The source notes that you may need more than 1,000 samples to push uncertainty below roughly plus or minus one percentage point, depending on the setup.
An LLM judge can help with the review step: it looks at the records inside each proposed identity, and decides how many distinct people they seem to represent. It doesn't magically correct the underlying IDR decision. What it does give you is a faster way to inspect outputs, compare model versions, and quickly find profiles that deserve human attention.
What the benchmark says about model accuracy and cost
The evaluation used pseudopeople, a tool from the Institute for Health Metrics and Evaluation at the University of Washington. It creates fictional records for residents in a simulated US town. The team then added realistic data errors and created 1,000 identities for the initial test: half were single-person cases with noisy records, and half were over-merges.
The models got a prompt describing the review task, explaining common data errors, and outlining where the records came from and the time span. Prompting mattered. Examples of typos, copied values, nicknames, and blank fields produced the biggest improvement across models. Source context helped frontier models by about one percentage point, and it didn't help the decision models in the same way.
Initial results: fast models were much cheaper
On the initial benchmark, Claude Opus 5.5 reached 99.0% accuracy at $3.86 per 1,000 identities, with an average response time of 2.2 seconds per identity. GPT-6 Luna (using the Decisions API) reached 97.3% at $0.10 per 1,000 and 0.17 seconds per identity. Jev reached 95.6% at $0.05 and 0.15 seconds.
Those numbers don't mean there's always a single winner. A decision model can be cheaper and faster, but its lower accuracy may not be acceptable if even a small number of mistakes have serious consequences. Meanwhile, a frontier model's accuracy might not be worth the cost for every routine review.
| Model setup | Accuracy | Time per identity | Cost per 1,000 |
|---|---|---|---|
| Claude Opus 5.5 | 99.0% | 2.2 sec | $3.86 |
| GPT-6 Luna, 70% cutoff | 97.3% | 0.17 sec | $0.10 |
| Jev, 60% cutoff | 95.7% | 0.15 sec | $0.05 |
The thresholds shown here were tuned for this benchmark. A model's confidence score isn't automatically a reliable probability, and the best cutoff depends on the trade-off between false positives and false negatives. Here, a false positive means labeling a profile as over-merged when it isn't; a false negative means missing an actual over-merge.
Harder cases changed the ranking
The team also tested trickier examples: people with similar names, twins, and parent-child pairs, alongside single-person records with disagreements in important fields. On the harder set, performance dropped across the board: Opus was at 95.1%, Jev at 89.1%, and Luna at 88.8%.
This is the practical warning: don't pick a model based only on its average score on easier examples. The confusing cases are the ones that decide whether your audit system actually helps. A benchmark should include the messy scenarios your data will produce - not just the obvious merges and obvious mismatches.
Why confidence calibration makes hybrid judging work
A hybrid evaluator runs a cheap model first, then escalates selected cases to a more capable model. The rule is simple: accept high-confidence judgments, and send the cases that fall into a chosen confidence range to the frontier model. That can cut average cost without running every case through the expensive model.
The key isn't only whether the first model is accurate. It also has to signal when it's likely to be wrong. A well-calibrated model tends to be less sure about its mistakes. That makes confidence a useful routing signal for deciding when to escalate.
Why Jev routed better than Luna
On the easier benchmark, Luna was more accurate than Jev when used alone. But the error analysis showed Jev's mistakes clustered closer to 50% confidence. Luna, meanwhile, made a noticeable share of mistakes in the 90-100% confidence range. Luna looked stronger at first glance, but its overconfidence made it a weaker routing signal.
That difference mattered more on the harder test. Jev's confidence shifted, but it stayed better calibrated than Luna's. As the data got harder, a routing policy could send more uncertain cases to Opus. Luna's high-confidence errors were less likely to be escalated, so the hybrid results fell further behind.
The cost and accuracy trade-off in practice
On the easier benchmark, sending Jev cases with confidence between 30% and 90% to Opus produced 98.8% accuracy. It escalated 22% of identities, cost $0.88 per 1,000, and averaged 0.63 seconds per identity. Opus alone achieved 99.0% at $3.86 per 1,000 and 2.2 seconds per identity.
On the harder benchmark, the Jev-plus-Opus setup reached 95.3% accuracy at $2.94 per 1,000, compared with Opus alone at 95.1% and $5.39. The savings were meaningful - but it wasn't a fourfold drop. The "more than fourfold" figure applies to the easier test's specific configuration; results will change as case difficulty and escalation rates change.
| Benchmark and setup | Accuracy | Cost per 1,000 |
|---|---|---|
| Easier: Opus alone | 99.0% | $3.86 |
| Easier: Jev, then Opus on 30-90% confidence | 98.8% | $0.88 |
| Harder: Opus alone | 95.1% | $5.39 |
| Harder: Jev, then Opus on 30-90% confidence | 95.3% | $2.94 |
The practical takeaway is that a hybrid evaluator's spending can rise when the first-stage model flags more "uncertain" cases. That's not automatically bad. If harder data triggers more review and prevents real mistakes, variable spend may be a better deal than a fixed cheap setup that silently misses difficult errors.
How to apply the approach to a production workflow
Treat the benchmark as a way to test, not as a model-selection recipe you can copy-paste. Costs, accuracy, and latency depend on your models, prompts, input sizes, and data distribution. A synthetic benchmark is great for controlled comparisons, but it doesn't guarantee how well the judge will perform on your actual customer records.
Build an evaluation set around your actual failure modes
Start by defining the decision clearly. For an over-merge audit, spell out what "one person" means, what the judge should output, and which evidence types count. Include examples of benign variation - like a nickname or a changed address - alongside examples that should trigger a "different people" judgment.
Then build a labeled evaluation set from representative cases. Mix in clean examples, ambiguous cases, and known failure patterns like shared addresses, similar names, and conflicting dates of birth. Keep a separate test set for final comparisons so you don't tune thresholds on the same cases you use to claim results.
Don't stop at overall accuracy. Track false positives and false negatives separately, because they create different operational consequences. Also measure the percentage escalated, cost per reviewed identity, latency, and performance on the harder slices. If a wrong merge could expose private information, your acceptable error balance may look different from a setup focused only on analytics.
Set the escalation policy with explicit trade-offs
A practical workflow can look like this:
- Run a fast decision model on each identity and keep its confidence score.
- Accept cases only when the confidence falls in a range validated on your evaluation set.
- Send uncertain cases - or high-impact ones - to a stronger model or a human reviewer.
- Log the first-stage output, confidence, escalation decision, second-stage result, and the final outcome.
- Review disagreements and update the benchmark when new failure patterns show up.
Don't assume one confidence interval will work for every dataset. The benchmark used settings like 30-90% for Jev, but those are experimental choices, not universal defaults. Tune thresholds based on the cost of each error type and on how strict your product needs to be.
Keep human review in the loop where it matters
LLM judges can make routine audits faster, but they can also share blind spots with the system they're evaluating. For example, if both the resolver and judge lean heavily on names and addresses, unusual household structures might get missed by both.
Use periodic human review on a random sample, plus targeted review of model disagreements, to surface those blind spots.
Treat changes to models and prompts as changes to an operational system. Re-run the evaluation set any time you change the identity resolver, judge, prompt, confidence threshold, or data sources. Watch escalation rate in production: a sudden jump can indicate data-quality problems or a harder mix of cases, while an unexpectedly sharp drop can mean the first-stage model has become overconfident.
And if you work under privacy rules like GDPR or US state privacy laws, decide what data can be sent to each model provider, how long it's kept, and whether the evaluation data includes personal information. Synthetic data can help early testing, but you still need real-world governance for the live workflow.
Conclusion
LLM judges can make identity-resolution audits faster, but it's not as simple as "cheap model vs expensive model." The real question is whether a fast first-stage model can identify the cases that deserve stronger review.
In this benchmark, Jev's better-calibrated confidence made it a more effective routing layer than Luna, even though Luna scored higher on the easier test when used alone. The hybrid Jev-plus-Opus setup got close to frontier accuracy at substantially lower cost, with the exact savings depending on case difficulty.
For real production use, build a benchmark from your own failure modes, track false positives and false negatives separately, and set escalation thresholds based on the cost of being wrong. Keep human review where uncertainty (and risk) still matters. That's what turns a promising model comparison into something you can actually audit.