Open ASR Leaderboard Adds Egyptian and Nigerian English
See what Egyptian and Nigerian English add to the Open ASR Leaderboard, how the new private conversational tests work, and what teams should learn from them.
Open ASR Leaderboard Adds Egyptian and Nigerian English
Egyptian and Nigerian English Put ASR Models to a More Realistic Test
Speech recognition benchmarks can look reassuring without answering the question that actually matters: does the model work for the people who will use it? Strong performance on familiar accents and carefully read speech doesn't automatically carry over to spontaneous conversation, regional pronunciation, or the little hesitations and false starts that happen in real life.
The Open ASR Leaderboard has added two private conversational English evaluation splits in collaboration with Liva AI: one for Egyptian English and one for Nigerian English. Each split contains a little over an hour of speech. They also extend an existing evaluation set that already includes private data covering multiple accents and speaking styles.
These additions are less about shifting a headline ranking and more about giving teams a better way to inspect model behavior. They add more natural conversational audio, broaden the accents represented, and make it harder to "game" performance against a fully public test set. Here's what changed, what the new data can (and can't) tell you, and how to use accent-level ASR results in product decisions.
What did the Open ASR Leaderboard add?
The leaderboard added two private evaluation splits supplied in collaboration with Liva AI. The Egyptian English split contains 1.34 hours of speech, and the Nigerian English split contains 1.55 hours. Both splits are described as conversational and spontaneous, with transcripts that preserve punctuation, casing, and disfluencies.
That matters, because a lot of ASR weakness only shows up outside clean, read-aloud audio. In real conversation, people change direction mid-sentence, use fillers, vary their speed, and lean on context. Preserving disfluencies in the reference transcript also helps avoid treating every hesitation as if it "doesn't count."
These new splits build on earlier additions to the Open ASR Leaderboard. The project previously included private data from Appen and DataOceanAI, covering scripted and conversational speech across five accents. It also added an Indian-accented English split from Voice Arena. With Liva AI, the leaderboard now adds two more accents and another data provider.
The announcement also describes the expansion in terms of Kachru's Three Circles, a common framework for discussing how English has different historical and social roles across regions. The announcement doesn't spell out enough to determine the exact placement of every accent or the full reporting layout. It does note that the Outer and Expanding Circles may be broken into smaller groups as more accents are added. That suggests the categorization is meant to evolve rather than function as a fixed map.
This is still an incremental update, not evidence that a single few hours of speech captures everyone in Egypt or Nigeria. The point is that it adds evaluation slices that were previously underrepresented, while keeping the comparison anchored in a larger leaderboard.
What is in each split?
| Dataset | Accent | Duration | Speaking style | Transcript conventions |
|---|---|---|---|---|
| Liva AI Conversational EG | Egyptian English | 1.34 hours | Conversational, spontaneous | Punctuated, cased, includes disfluencies |
| Liva AI Conversational NG | Nigerian English | 1.55 hours | Conversational, spontaneous | Punctuated, cased, includes disfluencies |
Why does accent coverage matter in ASR?
A speech recognition system doesn't hear "English" as one uniform signal. Pronunciation, rhythm, vocabulary, code-switching, recording conditions, and everyday conversational habits vary a lot from speaker to speaker and setting to setting. A model can look strong on a broad benchmark and still struggle with a specific user group.
And that gap can show up in real product work. If a voice assistant mishears names or instructions, users end up repeating themselves. If a transcription tool drops or mangles key words, the damage can flow straight into meeting notes, customer-service records, or accessibility features. In higher-stakes settings - healthcare and public services, for example - the cost of errors can be even higher. The application changes the details, but the underlying issue stays the same: an average score can hide uneven performance.
Accent-level evaluation helps make that unevenness visible. It also pushes teams toward better questions than "Which model has the lowest overall error?" They can ask whether performance changes across accents, speaking styles, and recording conditions that actually show up for their users.
Accent labels are useful, but they're not user profiles
An accent category is an evaluation label, not a full description of how a person sounds. Within the same country, speakers vary by region, age, language background, education, and context. People also adjust their speech depending on whether they're talking to a friend, dictating a message, or interacting with a support agent.
So adding Egyptian and Nigerian English is a step toward broader coverage, not proof that the evaluation reflects every speaker. The reported durations are small compared with the variation across either country. The splits can surface a model weakness worth investigating, but they can't tell you how every person in the population will experience the product.
There's also a transcript choice that affects how you interpret results. These splits retain casing, punctuation, and disfluencies. That can make the evaluation more sensitive to how well the system captures the full utterance - not just the key words. It also means you should interpret scores in light of what the benchmark counts as an error (and what it preserves in the reference).
For teams, the practical move is to treat public leaderboard results as a screening signal. If performance looks uneven across slices, investigate with data and tests that match your user base. If it looks strong, still validate it in the environments where the feature will ship. Either way, benchmark results don't replace product-specific evaluation.
What do private conversational test sets improve?
Public benchmarks make it easier to reproduce research comparisons and they create a predictable target for model development. Once the test data and scoring rules are widely known, teams can tune models against the benchmark - sometimes intentionally, sometimes indirectly. That can make leaderboard improvements arrive faster than real-world generalization.
Private evaluation data can reduce that pressure. If teams can't train against the hidden examples, the benchmark is less vulnerable to straightforward test-set memorization. Bringing in a new provider can also change parts of the data collection pipeline, which helps clarify whether a model's apparent strength depends on a particular recording setup or source.
Privacy isn't the same thing as quality, though. Private sets are harder for outsiders to inspect, reproduce, or diagnose. A score can still show that one model performed better under the benchmark's conditions, but it won't let the wider community audit every example. In practice, trust depends on more than keeping the audio hidden. Clear documentation - collection approach, transcript rules, split design, and scoring procedure - still matters.
Why conversational data changes the test
Scripted speech is great when you need controlled, repeatable audio. But people using voice products don't behave like actors reading the same prompt. They pause, restart, interrupt themselves, talk over background noise, and express the same intent in different ways. A spontaneous conversational split gives evaluators a window into some of those conditions.
The Liva AI description says its data comes from multimodal interactions through consumer products used by real people. It also claims control over both the collection environment and distribution, which lets it design products around gaps across domains, languages, modalities, and interaction types. That provenance suggests a different collection path than a narrowly scripted task, but the announcement doesn't provide enough detail here to evaluate sampling or consent for these specific splits.
There's a trade-off. Product-collected data can capture natural, high-context behavior, but it can also reflect the products, users, and situations that generated it. That's why independent test sets and multiple providers are useful: one source can add realistic speech, while a broader evaluation portfolio can help show whether models generalize beyond that source.
The balance is straightforward: private sets can make benchmark gaming harder, and public methods plus careful documentation can make results easier to understand. A credible evaluation program needs both protections and enough transparency for meaningful interpretation.
How should teams use these leaderboard results?
For an engineering or product team, a leaderboard is most useful during early model selection. It can help narrow options and flag obvious performance problems before you spend time integrating a system. It's not a replacement for testing the model on your own traffic, terminology, devices, and real-world failure costs.
Start by separating the benchmark question from the product question. The benchmark asks how systems performed on a defined set of audio and references. Your product question is whether the chosen system meets your quality, latency, privacy, and operating-cost thresholds for a particular workflow. They overlap, but they aren't the same thing.
A practical evaluation workflow
- Use the leaderboard to shortlist models. Look at overall results and any available accent-level slices. Don't pick a model from one aggregate score if your users' speech doesn't match the benchmark's dominant conditions.
- Check the evaluation conventions. Confirm what the transcripts preserve, how errors are scored, and whether the benchmark reflects spontaneous or scripted speech. A word error score can shift depending on normalization and transcription policy.
- Run a representative pilot. Collect or curate audio for your actual use case, with appropriate consent and privacy controls. Include the accents, devices, noise levels, and interaction patterns that matter for the product.
- Review errors, not just scores. Categorize failures such as missed names, dropped negations, punctuation mistakes, or weak handling of interruptions. The categories you care about depend on the system's purpose.
- Set release thresholds based on risk. A transcription error in a casual note-taking tool may be tolerable if users can correct it. The same kind of error may require human review in a regulated or consequential workflow.
- Monitor after launch. Real-world performance can change as users, microphones, and contexts shift. Track corrections and complaints where possible, and revisit the evaluation set when the product or user base changes.
Key takeaways
- The leaderboard now includes 1.34 hours of Egyptian English and 1.55 hours of Nigerian English conversational speech.
- The new splits are private and preserve punctuation, casing, and disfluencies in their transcripts.
- More accent slices make hidden gaps easier to spot, but these small datasets don't represent every speaker in either country.
- Use leaderboard results to shortlist models and guide further testing, not as a guarantee of production performance.
- Private data can reduce direct benchmark tuning, but transparency about methods is still needed for trustworthy interpretation.
This requires effort, but it avoids a common and expensive mistake: adopting a model because it wins a benchmark that doesn't resemble what you're shipping. The model you want is the one that meets your requirements under the real conditions you can reasonably expect - paired with a clear plan for the errors you can't eliminate.
Conclusion
Adding Egyptian and Nigerian conversational English gives the Open ASR Leaderboard a more useful view of how recognition systems perform across accents and natural speaking styles. The private splits also support the broader goal of making evaluation less dependent on a small set of familiar, public tests.
The limits matter as much as the additions. A few hours of audio can reveal gaps; it can't settle whether a model will work for an entire user population. Treat the leaderboard as a starting point, focus on the slices that match your product, and validate against the speech and risks you're most likely to encounter.