Speech recognition is easy to underestimate. A transcript can look fine and still be unusable - speakers blur together, technical terms get mangled, numbers show up in the wrong format, and multilingual meetings often need translation, not just a word-for-word dump. Fixing those issues downstream is extra work, and it's also where new errors creep in.
NVIDIA's Nemotron 3.5 Transcribe tries to handle more of that up front in a single open-weight model. It combines transcription with speaker attribution, translation and audio question answering, and it lets developers steer output using natural-language instructions. The accompanying Granary v2 release brings multilingual data manifests and a reusable curation pipeline, and NeMo Speech supports fine-tuning when you need domain-specific behavior.
The real question isn't whether one model can do everything. It's whether your team can use prompting for everyday variations, fine-tune when the domain genuinely demands it, and validate results on the recordings you actually handle. This is what the release offers, what the benchmark numbers do (and don't) tell you, and how to decide if it fits a production workflow.
What does Nemotron 3.5 Transcribe actually do?
Nemotron 3.5 Transcribe is an open-weight model for recorded audio. It supports transcription in 40 languages, translation to or from English across 31 languages, and speaker attribution in 13 languages. It can process recordings up to an hour long and label up to eight speakers.
A big product decision here is that several speech tasks come in one checkpoint. That can reduce the wiring work you'd otherwise do across separate transcription, diarization, translation, and analysis services. Still, you can't assume every task behaves the same way: language coverage varies, and audio question answering was trained in English.
One checkpoint, several speech tasks
With one meeting recording, you can get different outputs by changing the instruction. For example, you can request a transcript with speaker labels, an English translation, or answers to questions about decisions made during the meeting. Word-level timestamps are supported too, which helps when the text needs to stay aligned with playback.
This matters for products like meeting software, classroom recordings, customer-support review, and interview archives. Agora, for example, used word-level timestamps in a classroom demo that shows a transcript next to a recorded lesson. The practical win isn't just fewer model calls - it's that the output lines up more directly with what users see and hear.
There are real limits, though. Nemotron 3.5 Transcribe is built for recorded audio, not live streaming. A real-time product still has to deal with capture, latency, partial results, and interruptions around the model. Treating an hour-long recording model like a drop-in streaming engine would be a mistake.
Prompting is a control layer, not a guarantee
Natural-language instructions can specify language, script, transcript style, or a list of relevant terms. A medical transcription prompt might call out terms like HbA1c and metformin and request those spellings only when the audio supports them.
That last part is key. A vocabulary hint can improve recognition, but it can also nudge the model toward a term that wasn't actually spoken. Prompts should guide formatting and context, not override what's in the audio. Teams should spot-check names, speaker changes, and answer quality against the recording - especially when errors affect care, compliance, or customer records.
Prompting is a good first, low-cost step because it changes behavior without changing the model weights. It's most helpful when your main issue is output convention or a small set of terms. If the same kinds of errors show up across speakers and recordings, you're probably looking at an underlying recognition gap that needs representative training data, not just a longer prompt.
How strong are its benchmark results?
NVIDIA reports strong results across multiple public evaluations, but each benchmark reflects a different setup. So the numbers are useful evidence, not a substitute for testing on your audio, languages, and error tolerance.
Transcription accuracy
On the Hugging Face Open ASR Leaderboard, Nemotron 3.5 Transcribe ranks first among open-weight models in the English track, with a reported average word error rate (WER) of 5.33%. NVIDIA also reports a mean WER of 4.62% across seven languages in its multilingual comparison and 10.98% on the long-form comparison.
Artificial Analysis reports a 2.8% AA-WER, tying for first among open-weight models in its speech-to-text evaluation. Its AA-AgentTalk result is 1.8% WER on a held-out voice-agent speech set. NVIDIA also describes the model size as about four billion active parameters per token (out of roughly 30 billion total), using a hybrid Mamba-Transformer mixture-of-experts language model alongside its speech encoder.
A practical caution: don't compare these figures as if they came from one shared test. Datasets, normalization rules, recording conditions, and scoring all differ. And WER treats all word errors equally, even though a wrong drug name can matter far more than a missed filler word. If you're making procurement or product decisions, reproduce the evaluation on audio similar to your users' and score the errors you actually care about.
Speaker attribution, overlap and throughput
Diarization gets harder when people interrupt each other or when multiple speakers overlap. NVIDIA reports that Nemotron 3.5 Transcribe had the lowest diarization error rate among 14 systems on David AI's 21-language benchmark, at 18.26%. It also ranked first on VoiceArena's English full-session comparison, with 32.4% concatenated minimum-permutation WER across sessions with overlapping speech and two to eight speakers.
These results are a good reason to test it on interviews, meetings, and panel recordings. They don't mean every speaker label will be right in every situation. Diarization scores summarize speaker-time errors across a dataset; they don't guarantee clean attribution on your specific call with your specific microphones and voices.
On performance, NVIDIA reports processing 100 hours of audio in under three minutes on an RTX PRO 6000 Blackwell Server Edition GPU, using a forthcoming NVFP4 quantized version and high concurrency on 30-second clips. That's a throughput benchmark, not a promise for every deployment. Real cost and latency depend on hardware, batch size, audio duration, and serving configuration. NVIDIA notes that the checkpoint can be served with vLLM, and that local deployment options and quantized versions were described as forthcoming in the release.
How does Granary v2 support adaptation?
A promptable model still needs training examples that clearly separate what was said from how the text should look. Granary v2 addresses that with multilingual data manifests and a reusable curation pipeline built with NVIDIA NeMo Curator.
The public manifest covers 463,062 hours across 48 languages and 16 source corpora. These numbers describe the released dataset manifests, not the full training mixture for Nemotron 3.5 Transcribe or the sampling weights used during training. Source audio remains under each corpus's original license, so teams need to check terms before reusing anything.
Multiple written targets for the same speech
Granary v2 creates three transcript forms for a recording: text-normalized verbatim, inverse text normalization with disfluencies, and inverse text normalization without disfluencies.
For the spoken phrase "I paid, uh, twenty-five euros," the targets could preserve the spoken wording, write the amount as "€25" while keeping "uh," or produce "I paid €25" with the filler removed.
That choice has product consequences. A legal or research transcript might need disfluencies preserved; a meeting summary often doesn't. If an application expects a specific number, currency, or brand-name convention, you should define which target you want instead of assuming one generic transcript format will always work.
The pipeline also generates task labels like short audio captions, translations, speech-grounded question-and-answer pairs, code-switching restorations, and contextual ASR terms. A quality flag and provenance metadata help users understand what was produced and how it was curated.
A reusable four-stage curation process
The pipeline has four stages:
- Audio preparation: segment recordings, track speakers, and identify acoustic events and language. Granary v2's source data was already segmented for this release, so this stage was skipped there.
- Transcription: route each language to primary and fallback ASR systems, with checks intended to catch hallucinated output. Human references can help retain trusted spellings while adding spoken forms and disfluencies.
- Task-label generation: derive transcript variants, then check factors like character rate, hallucination patterns, and agreement with references. Accepted records can receive contextual terms and translations.
- High-fidelity filtering: select cleaner audio for use cases that need higher recording quality. In one example, a VoxPopuli subset dropped from 354,000 hours to 111,000 hours across 22 European languages.
For a product team, the process matters as much as the dataset. You can build on the public manifests or run the curation pipeline on recordings you're authorized to use. Either way, quality checks and explicit targets matter - otherwise you can end up teaching a model inconsistent conventions at scale.
When should you prompt, fine-tune, or choose another system?
Use the least expensive intervention that reliably fixes the errors you actually see. Start with prompting when the model understands the speech but needs clearer instructions about language, script, terms, or formatting. Fine-tune when errors are systematic, domain-specific, and you have enough representative, labeled data. Consider another system when latency, streaming support, language coverage, privacy controls, or measured accuracy don't match your requirements.
A practical adaptation workflow
A careful evaluation helps teams avoid a common trap: mistaking a prompt improvement on a small sample for a real production win.
- Define the failure precisely. Separate recognition errors from formatting errors, speaker-label errors, and incorrect answers to audio questions. A single aggregate WER can hide the issue that matters most.
- Build a representative development set. Use real speakers, accents, microphones, and environments. Keep a separate held-out set so prompt and checkpoint choices don't leak into final evaluation.
- Try a constrained prompt. Specify language, script, and transcript conventions. Add a short term list when it helps, and tell the model not to insert suggested terms unless the audio supports them.
- Measure the right errors. Track overall WER alongside domain terms, numbers, speaker attribution, and semantic accuracy. Keep a general-speech test to catch regressions outside your target domain.
- Fine-tune only if the remaining error justifies it. Use explicit written targets and representative speech. Choose checkpoints based on decoded development results, not training loss alone, then evaluate once on held-out speakers.
NVIDIA's medical ASR case study shows the distinction. On Eka Care's Indian medical speech benchmark, a prompt specifying English and the Latin alphabet reduced full-benchmark WER from 30.8% to 23.8%. Fine-tuning brought it to 11.8% on the full set. Because that full set included clips used to choose prompts and checkpoints, the more trustworthy result is the held-out-speaker evaluation: WER fell from 22.44% for the best-prompt baseline to 12.17%, a 45.8% relative reduction.
That adaptation combined synthetic medical speech, accented English, and medical recordings. NVIDIA reports the workflow took about 14 GPU-hours on one GB300. On held-out speech, keyword WER dropped from 9.2% to 6.5%, and semantic WER dropped from 10.0% to 7.6%. The adapted system also outperformed the medical-specialist and general-purpose models included in the study.
There was a trade-off: LibriSpeech test-clean WER rose slightly from 1.54% to 1.69%. The medical specialist still did better on exact matches for some brand-name drugs and diagnostics. That's why a domain adaptation result should be judged against both your target-domain needs and your general-speech retention - not a single headline percentage.
For your own project, NVIDIA suggests keeping at least 100 real, labeled clips for checkpoint selection. That's a starting point, not a universal rule. Higher-risk or more variable domains may need substantially more data, and synthetic speech can't fully replace real accents, microphones, and speaking styles.
What should businesses take from this release?
The business case isn't just "one model replaces four." A combined checkpoint can reduce integration work and make output conventions easier to test, but your team still owns serving, validation, data rights, and how failures show up for users. Open weights give you more control than a hosted API, but they also shift infrastructure and maintenance responsibilities to you.
For product teams, the strongest fit is a recorded-audio workflow where speaker labels, multilingual transcription, translation, or timestamps create real user value. Meeting archives, classroom replays, interview tools, and specialized dictation are natural candidates. Live voice agents need a separate latency evaluation, and regulated applications need clear policies for review, retention, access, and correction.
The data pipeline brings another operational question: do you have permission to use the audio, and can you trace the source and the target conventions? Granary v2's manifests and provenance fields are helpful, but they don't automatically grant rights to the source recordings. For customer or patient audio, governance and consent have to be part of the system design - not added later.
Key takeaways
- Use prompts to control output conventions before changing model weights.
- Validate benchmarks on your own languages, speakers, and recording conditions.
- Fine-tune for repeatable domain errors, and keep held-out speakers for evaluation.
- Track costly errors - such as names, numbers, and speaker attribution - separately from aggregate WER.
- Treat deployment, data rights and human review as part of the product, not separate implementation details.
Conclusion
Nemotron 3.5 Transcribe pairs speech recognition with speaker attribution, translation, and instruction-driven output, while Granary v2 makes transcript conventions and data curation more explicit. The reported evaluations are a reason to test it - but they don't remove the need to validate against your own recordings and your own error costs.
Start with a bounded evaluation, use prompts for formatting and context, and fine-tune only when real data shows a persistent domain gap. That sequence keeps adaptation tied to product needs, and it makes it easier to judge whether the extra model complexity is actually paying off.