Indic Pretraining Data: 2.8T Tokens for LLMs ⚡ (2026)
Discover NVIDIA & AI4Bharat’s Indic pretraining data (2.8T tokens): multilingual LLM data generation pipeline, training results, and lessons—read more.
Building Indic AI at Scale: What a 2.8 Trillion-Token Corpus Changes
A multilingual model can be big, technically capable, and still disappoint the people it's supposed to help. One obvious culprit is data: many Indian languages have far less high-quality training material online than English. That gap shows up fast in what teams can train, test, and ultimately deploy.
NVIDIA, IIT Madras, and AI4Bharat have announced Nemotron-Pretraining-Indic-v1 - a corpus of roughly 2.8 trillion tokens spanning 22 Indian languages. They're also sharing the data-generation recipe: a GPU-accelerated pipeline that translates English source documents, checks quality, repairs some weak outputs, and produces selected romanized text.
The headline number is impressive, but the more useful takeaway is operational: synthetic data isn't "translate lots of documents and hope." It takes clear provenance, quality gates, language- and script-specific handling, and evaluation that makes the system's limits obvious. This article walks through what's in the corpus, how the pipeline runs, what the reported results do (and don't) show, and how teams can use the approach without mistaking volume for quality.
Why this corpus matters
The data gap is a product problem, not just a research problem
A language model's usefulness depends partly on what it has seen during training. When one language dominates the available data, quality can end up uneven: the model might be fluent in English but struggle with terminology, script conventions, or everyday phrasing in another language. That shows up in real products - education, customer support, public services, and software features that need to work reliably for more than one audience.
The announcement ties the corpus to the 22 languages listed in India's Constitution, described as the first languages of about 97% of the population - nearly 1.2 billion people (2011 Census). The message is clear: language support isn't a "nice to have" when it affects whether a large group can actually use a product.
Still, more tokens don't automatically mean a better product. Teams need to check whether the training data matches their use case, whether technical content survives translation, and whether improvements hold up across languages (not just in aggregate). In production, a model that answers more questions but becomes less reliable for one target language is not an unqualified upgrade.
Synthetic data can expand coverage, with a boundary
Nemotron-Pretraining-Indic-v1 is mostly generated by translating curated English content into Indic languages. The sources include 15 Nemotron-CC-v2 subsets, with web text, question-and-answer material, mathematics, code, and other specialized content. In practice, that choice is aimed at bringing more technical and task-oriented material into languages where comparable training data is harder to find.
That can help - but translated text isn't the same as naturally authored writing. Translation can move knowledge across languages, but it doesn't automatically capture local context, cultural references, community-specific vocabulary, or the way people actually phrase things in everyday conversation. The project itself points to combining the corpus with appropriately sourced natural Indic text.
So for AI teams, the sensible framing is coverage expansion, not replacement. Synthetic data may help a model explain a math concept in a target language or answer a translated question. It shouldn't become the only representation of that language in the training mix.
What is in the 2.8 trillion-token corpus?
Twenty-two languages and 34 language-script configurations
The release reports about 2.8 trillion tokens across 22 Indian languages. The inventory lists 2,601.0 billion native-script tokens and 224.6 billion romanized tokens - about 2,825.6 billion total tokens. Token counts use the Gemma-4 tokenizer, so they're dependent on the tokenizer rather than a universal measure of "corpus size."
The configurations fall into two groups:
- Native-script and romanized data: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu.
- Native-script data only: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi.
"Native-script" refers to the writing system, not where the text originally came from. Much of the material is translated from English and rendered in the language's own script. Romanized data is Indic-language text written with the Latin alphabet; it isn't English text. For the twelve languages with two script variants, the romanized material is a subset of accepted native-script translations.
This distinction matters when building a training mixture. Two records can represent the same underlying information in different scripts, so total token count isn't the same thing as the number of unique source documents. And the corpus inventory shouldn't be confused with a model's training-token budget - those are different quantities.
Provenance is part of the dataset design
The workflow keeps source identifiers, so generated records can be traced back to the documents they came from. That's more than just convenient metadata. Provenance helps with auditing examples, checking for duplicated processing, and understanding where a bad translation likely originated.
It doesn't fix every data-quality issue. Source-ID checks can flag processing duplicates, but they don't provide semantic deduplication - two different documents can still cover very similar material. Teams should keep that in mind when building validation or test splits. If translated versions of the same underlying source end up in both training and evaluation, benchmark results can exaggerate generalization.
The corpus and pipeline are intended for open release. If you're planning to use them in a commercial product or research run, it's worth checking the dataset card, license, available language configurations, and revision details. Dataset availability and terms can change, and pinning a specific revision helps reproducibility.
How the open data-generation pipeline works
The core engineering choice is to treat generation as a chain of stages rather than a single "translate everything" call. The pipeline uses NVIDIA's NeMo Data Designer and language-specific workflows to translate documents, assess outputs, and apply targeted processing. The project shares the recipe so other teams can adapt prompts and checks to their own sources.
Twelve languages: translate, assess, repair, then romanize
For twelve languages, the workflow uses Gemma-4-31B-it for document-level translation. The team ran comparisons on complete documents, with attention to both translation quality and the preservation of technical material. Prompts separate "prose to translate" from content that should stay intact - equations, code, URLs, and formatting.
That same model then assesses each translation against its English source. The rubric draws on the Multidimensional Quality Metrics framework, covering meaning, fluency, terminology, style, and formatting. A translation must score at least 75 out of 100 to pass. This is a rubric score, not a claim that translation accuracy is "75%."
If a translation comes in below the threshold, the workflow can repair it using feedback from the assessment, then reassess against the same cutoff. The reported process recovered 438.4 billion tokens across native-script and romanized data; those tokens are included in the overall corpus total. Sampled human review and tests that introduced deliberate errors helped identify omissions and residual English, leading to improved prompts and checks.
Romanization happens only after an accepted native-script translation. The system converts a subset into Latin script and checks the result against the native-script version. Separating translation from script conversion is a deliberate quality choice. In an initial human comparison, the two-step setup produced fewer hallucinated additions and less residual English than generating romanized text directly from English.
Ten languages: adapt the method to the language
The remaining ten languages use a different workflow, reflecting issues the team saw in long-document quality and script stability. Translation uses Indic Translate, a document-level model for English and the 22 Indian languages built by Bodhan AI (a nonprofit center for AI in education incubated at IIT Madras). Gemma-4-31B-it then assesses translations against the English sources.
Eight languages use a whole-document approach. Longer Santali documents are translated in overlapping windows and reconstructed. For Manipuri, the workflow translates into Bengali script first, then converts the result into Meitei Mayek. These outputs are filtered with the same 75-out-of-100 threshold, but this workflow does not include feedback-based rewriting and produces native-script data only.
That difference is important. A pipeline that works cleanly for one language and script may not transfer well to another. If you're building your own data workflow, treat language and script as design variables - not something you can slap on after translation. A single global configuration might be simpler to operate, but uneven results can hide until evaluation.
What the results mean for model builders
The project reports two kinds of evidence, and you shouldn't treat them as interchangeable.
Continued pretraining: a measured stage, not proof of one dataset's effect
During continued pretraining for Nemotron 3.5 Super VL, the configured budget was 100 billion training tokens. 10% (10 billion tokens) was allocated to a mixture of synthetic and natural Indic text. The remaining 90% kept the relative proportions from the preceding training mixture.
The report compares checkpoints immediately before and after this Indic continued-pretraining stage. It shows gains on all five reported Indic knowledge and reasoning benchmarks: MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K. Reported gains range from 4.98 to 9.82 percentage points. The GSM8K evaluation excluded Odia because it exceeded the serving context limit.
These results support the overall continued-training recipe. They don't isolate the contribution of this synthetic corpus from natural Indic data, other training changes, or interactions between them. If you're deciding whether to adopt the dataset, that checkpoint comparison tells you about the stage - not a controlled estimate of one data source's effect.
The companion Indic report also looks at source mixtures, the synthetic-data pool size, and script coverage for first-pass accepted data in the twelve-language group. It calls evaluation of the full corpus - including recovered rewrites and the other ten languages - a next step for controlled studies.
Full model comparisons include more than continued pretraining
A separate comparison looks at the released Nemotron 3.5 Super VL model versus the previous Nemotron 3 Super generation. The newer model scored higher on five of six task benchmarks shown, spanning knowledge, reasoning, mathematics, code, and translation. Its IFEval 1.5 instruction-following score decreased by 2.04 points.
On top of that, the model improved on all four IndiVibe dimensions: usefulness, fluency, language and script fidelity, and appropriate verbosity. IndiVibe is an open benchmark from Sarvam AI for assistant-style prompts in Indian languages. The full-model results reflect the entire training pipeline - continued pretraining plus post-training - so they shouldn't be credited to the corpus alone. IndiVibe's conditional preference scores also aren't direct head-to-head win rates.
For a product team, the takeaway shouldn't be "this dataset guarantees a lift." It should be: measure language quality alongside task capability, English performance, and instruction following. Gains in one area can come with regressions in another. Test trade-offs against the tasks and users you actually care about.
Limitations, responsible use, and practical next steps
Start with a bounded experiment
This release is most useful as a resource to test, not a drop-in answer for multilingual training. A sensible first experiment looks like this:
- Choose target languages and scripts. Start from your audience and product requirements. Don't assume native-script and romanized data behave the same way.
- Inspect records before training. Sample documents from the relevant configurations. Check technical terms, equations, code, tables, formatting, and any residual English.
- Hold the token budget constant. Compare mixtures at a fixed budget so differences are interpretable. Track the shares of natural vs. synthetic data and native-script vs. romanized data.
- Evaluate per language. Track both task performance and language quality, not just one aggregate score. If English matters for your product, include English capability retention.
- Split by source lineage. Keep related translations and script variants from the same source out of separate train/test partitions when you can.
- Pin data and code revisions. Record the dataset revision, prompts, filtering thresholds, tokenizer, and training mixture. Otherwise, it may be impossible to reproduce results later.
Hugging Face Datasets supports streaming, which makes it easier to inspect samples without downloading the full corpus. Once the release is available, the dataset card should provide the repository ID and configuration names. The announcement includes examples like hi-Deva-IN for Hindi in Devanagari and hi-Latn-IN for romanized Hindi.
Adapt the generation recipe with native-language review
The open-source Nemotron Bharat translation plugin for NeMo Data Designer is meant to expose the workflow's translation and evaluation prompts, optional feedback-based rewriting, document windowing, and script remapping. The right starting point is a small sample of your own source documents - not a full production run.
Check whether the system preserves what matters in your domain. A legal document, a math explanation, and a software manual can fail in different ways. Native-language reviewers can catch unnatural phrasing or incorrect terminology that automated scoring may miss. Also keep candidate generation and acceptance separate: a pipeline can produce candidates and scores, but your team still needs to decide what's safe and useful to train on.
The project also points to an evolving Indic-LLM-Eval-Suite developed by AI4Bharat at IIT Madras with support from NVIDIA. Better benchmarks matter because task accuracy alone doesn't fully capture whether a model communicates naturally or respects the target script. For related details, see the original NVIDIA and AI4Bharat announcement and the linked technical reports.
Keep the ethical and operational boundary visible
Translation can make existing sources accessible in more languages, but it doesn't make the underlying source complete, representative, or appropriate for every use case. The corpus can still contain translation errors, and review coverage can vary across languages. The ten-language workflow in particular likely needs broader native-speaker validation, per the announcement.
There's also a practical trade-off: generating and validating data costs compute, engineering time, and review capacity. Open tooling can reduce some of the cost of building a pipeline, but it doesn't remove the hard part - deciding what belongs in the training set and proving the resulting model works for your users. If you can't evaluate a language well, scaling data generation before fixing evaluation gaps can create confidence without evidence.
Use the corpus alongside naturally authored Indic text and evaluate the final mixture for your intended task. It's a less dramatic claim than "more tokens solve the language gap," but it's also the one you can responsibly test.
Conclusion
A 2.8-trillion-token Indic corpus is a meaningful addition to open resources for multilingual model development. Its most reusable contribution may be the end-to-end generation recipe: document-level translation, explicit quality assessment, targeted repair, script conversion, and source traceability.
The numbers aren't a substitute for controlled experiments. Start with the languages and scripts your users actually need, inspect samples, compare training mixtures at a fixed budget, and evaluate both task capability and language quality. Combine synthetic material with natural Indic text, and keep source lineage in mind when designing evaluation.
That's how teams can turn a large corpus into evidence about their own models and products. Explore the project announcement and linked resources, review the dataset card and license, and run a small, reproducible training mixture before scaling up.