SLM Vocabulary Optimization: Cut Inference Costs
Learn how cost-aware vocabulary specialization cut inference costs by 77% in a Portuguese essay grader—and when the approach makes sense for your SLM.
Can a Better Vocabulary Make Small Language Models Cheaper to Run?
Small language models are often tuned by changing their weights: quantize them, fine-tune them, or pick a smaller base model. There's another lever people miss, though: the vocabulary the model uses to break text into tokens.
A general-purpose tokenizer has to cover lots of languages, topics, and formats. That flexibility is great for an assistant that's meant to handle "everything." For a system that does one job over and over - say, grading Portuguese essays or generating structured JSON - that flexibility can become overhead.
If the tokenizer splits common domain phrases into many tokens, responses can get longer. And at generation time, the model still has to score a large number of token options it probably won't use.
Dharma-AI says its VocabX method cut measured inference cost by 77% in a production Portuguese essay grader, without changing the base model or training recipe. That's encouraging - but it's not a guarantee you'll get the same gain. It depends on the task, the output format, how you serve the model, and whether quality holds up after the change.
This article breaks down the cost model behind the result, how VocabX specializes a vocabulary, what the reported experiments do (and don't) show, and how to judge whether this is worth testing in your own system.
Why a general vocabulary can waste inference budget
A tokenizer decides how text is split into tokens before a language model processes it. General-purpose tokenizers are designed to handle many languages and use cases, which makes sense for models expected to cover everything from everyday questions to code and legal documents.
A specialized system looks different. If it runs in one language, one domain, and a stable response format, the general vocabulary may represent the most common text in an inefficient way.
Fragmentation makes repeated outputs longer
Take an automated grader that returns a JSON assessment for every essay. It repeatedly uses the same field names, competency labels, punctuation, and evaluation phrasing. If the tokenizer chops those recurring spans into multiple pieces, the model has to regenerate those pieces on every request.
You'll see the same pattern in other products:
- A contract assistant that repeatedly encounters standard clauses and defined terms.
- A code model working with recurring identifiers, syntax patterns, and project-specific names.
- A claims-processing model that outputs a fixed set of fields and status labels.
- A medical documentation system with consistent headings and phrases.
Token count matters, but it isn't the whole picture. A token in the prompt is not always priced the same as a token the model generates.
Generated tokens have a different cost profile
Inference is often described in two stages. In prefill, the model reads the input context. In decode, it generates the response one token at a time. Each decode step runs the model again, and that repeated work can hit limits like memory bandwidth - especially for small models served in ways that don't hide latency well.
Exact cost depends on hardware, batch size, context length, and serving software. In Dharma-AI's essay-grader measurements, they estimate that one generated token cost 7.4× more than one input token. That ratio is specific to their setup, not a law you can apply everywhere. Still, the direction is important: shortening output can matter more than shortening input.
So you can't evaluate a tokenizer change by prompt length alone. If a method saves many tokens in a long prompt but hardly reduces generated output, its impact may be smaller than it looks. A cost-aware approach weights token savings based on where they occur, not just how many tokens you removed.
A large vocabulary also has a recurring decode cost
The output layer - often called the language-model head - scores possible next tokens. It contains a row for every vocabulary entry. At each generation step, the model uses this layer to score candidates across the whole vocabulary.
So vocabulary size can affect more than tokenization. A larger vocabulary means more rows in a scoring matrix that gets used repeatedly during generation. For smaller models, that extra work can be a meaningful share of the compute (and the amount of data moved) during decode. Tokens you almost never use still sit in the output layer and still get considered.
There's a trade-off. Adding domain-specific tokens can reduce the number of tokens needed to express a phrase, but a bigger vocabulary can increase the work of scoring possible outputs. Vocabulary design is therefore an optimization problem, not a simple contest to cram in as many useful tokens as possible.
If you want the serving-side constraints too, see our guide to LLM inference cost optimization. The key question isn't just "How many tokens can we remove?" It's "Which tokens can we remove, from which part of the request, without breaking what the product needs?"
How cost-aware vocabulary specialization works
VocabX is built to adapt a model's vocabulary to a domain and task. In the reported grader experiment, Dharma-AI kept the same Qwen3-4B base model, training data, training recipe, and seed across versions. The tokenizer was the variable under test.
The method differs from approaches that mostly choose tokens based on how often text appears in a corpus. Frequency helps, but it doesn't tell you how expensive a token is at serving time. If a repeated phrase shows up in generated output, compressing it can be more valuable than compressing a phrase that appears often in input.
Optimize for serving cost, not token count alone
VocabX assigns different value to savings depending on where they happen in the request: system prompt, user message, or model response. In the essay-grader setup, they treat a saved response token as worth 7.4× a saved input token. Candidate vocabulary changes are selected with those relative costs in mind.
That's the practical shift: the "best" vocabulary for a retrieval-heavy system with huge prompts may not be the best vocabulary for an app that produces long, repetitive reports. The right choice depends on real traffic patterns, not just the training corpus.
Improve both subword splits and repeated spans
VocabX works at two levels:
- Subword vocabulary: It changes how domain-specific words are split into smaller pieces.
- Macro tokens: It can represent a recurring multi-word span (for example, a repeated line or a chunk of a JSON structure) as one token.
The macro-token idea is most useful when the repeated unit is bigger than a single word. In the grader, repeated response structure - especially the JSON envelope and competency labels - was a noticeable source of avoidable output tokens. A subword-only tokenizer might help with individual word splits, but still need several tokens for a whole repeated line.
To find candidates, VocabX uses five automated mining processes aimed at different types of repetition: long words, lines, expressions, sentences, and punctuation-heavy structures like JSON. The paper preview says the punctuation/structure miner contributed a lot across their tested corpora. That's not proof you should add macro tokens everywhere - it's a reminder that the shape of your output matters when deciding what to compress.
Avoid treating unseen candidates as worthless
Vocabulary selection has to estimate how much each candidate will save once you account for other candidates already chosen. If you recomputed the entire corpus for every possible choice, it would be too expensive, so sampling is common.
But sampling creates a problem for rare candidates. A phrase might appear enough times in the full corpus to matter, but not show up in a small sample. If you treat "not in sample" as zero value, you effectively exclude that candidate.
VocabX separates a candidate's total potential savings (countable across the full corpus) from the part already covered by tokens you've already selected. For candidates the sample doesn't show, it estimates the uncovered portion instead of assigning them no gain. In the reported contracts-corpus experiment, this improved token reduction from 44.7% to 52.8% using the same sample and budget.
That detail matters in practice: better candidate selection isn't just about chasing frequent phrases. It's also about avoiding a sampling strategy that systematically misses useful ones.
What the reported results show
The headline result comes from a Portuguese essay grader built on Qwen3-4B. The system takes an essay, a prompt, and supporting texts, then returns a JSON assessment with a grade and a comment for each competency.
Dharma-AI reports training seven versions of the grader with the same base model, data, recipe, and seed, changing only the tokenizer. Each version ran on the same 500 requests on an RTX 5090 using vLLM. They used runtime measurements to estimate inference cost.
In that test, the average response dropped from 350 tokens to 101. They report measured cost falling by 77%. The benchmark score (combining grading accuracy with the ability to filter out off-topic or rule-breaking essays) rose from 0.703 to 0.810.
Those numbers are useful because the comparison keeps several major variables constant. They also have limits: it's one application, one test set, and one hardware + serving configuration. It doesn't prove that any model will reduce inference cost by 77% just by changing its tokenizer. The VocabX paper was described as under review at the time the source article was published, so the authors explicitly note results may shift.
Compression can damage the behavior you care about
The grader experiment also shows why "fewer tokens" isn't automatically the same as "better product."
An early version fused a competency's grade together with surrounding JSON into a single macro token. That was more compact, but the benchmark score fell to 0.689 - below the production grader baseline of 0.703.
Then they separated the grade into its own token. That added two tokens per response, but the score increased to 0.810. They also reported a stronger cost reduction: 77.0% for the separated version, versus 73.3% for the more aggressively fused version.
The bigger product lesson is a practical one: preserve the parts of the output where the model has to make a decision. If a macro couples a label or grade with formatting, it can make that distinction harder for the model to express. In other words, the "smallest" representation isn't always the best one.
Results across other domains are encouraging, not conclusive
The source reports comparisons against three reference approaches across five public domains - contracts, medical papers, scientific papers, news, and Python code - and three model families: Qwen3, Llama-3.1, and Gemma-3. Under the tested vocabulary limits and evaluation setup, VocabX compressed more than the strongest reference method in all 15 combinations.
The pruned VocabX version also led on estimated inference cost in those combinations. On contracts, they report a 62.6% token reduction with 113,426 vocabulary entries, compared with 61.8% for the best in-place BPE configuration, which used 207,803 entries.
These results support trying vocabulary adaptation in specialized systems. They don't remove the need to test quality, latency, and cost on your own workload. For a broader view of small language models in production, focus on your actual traffic and output patterns instead of assuming benchmark results transfer unchanged.
When vocabulary specialization makes business sense
Vocabulary specialization is most promising when your model has a narrow job, sees stable formats, and generates repeated domain-specific text at meaningful volume. In that case, the savings can show up again and again across requests.
If you only use a general-purpose assistant occasionally, the engineering and evaluation work may outweigh the benefit.
A practical checklist is basically four questions:
- Is the task genuinely narrow? If the model serves one language, domain, or workflow, it's a better candidate than a general assistant that needs broad coverage.
- Does the model repeatedly generate the same structures? Structured JSON, stable fields, templated reports, and recurring technical terms create opportunities for specialization.
- Is generation a material share of your serving bill? If prompts dominate cost, compressing output may not move the total much. Measure input and output cost separately.
- Can you validate behavior? Check quality, formatting reliability, latency, and failure rates. Don't trust token count alone.
Run a controlled deployment test
Don't start by swapping the production tokenizer and hoping for the best. Do a reproducible comparison first:
- Capture representative requests, including both typical inputs and edge cases.
- Record input tokens, output tokens, latency, throughput, and serving cost separately.
- Train or adapt the model with the intended tokenizer and keep other variables fixed as much as possible.
- Evaluate task outcomes, including structured-output validity and any safety/policy checks relevant to the product.
- Test with the same inference server, hardware, batching, and context lengths you plan to use.
- Estimate the maintenance cost if you have to regenerate the vocabulary as the domain or output schema evolves.
That last part is easy to overlook. A specialized vocabulary is tied to the languages and formats the model actually encounters. If you add new output fields, expand to another language, or change workflow details, you may need to re-evaluate (and possibly rebuild) the vocabulary and model. Specialization can create leverage, but it also creates dependency on the assumptions behind it.
For teams already running high-volume SLMs, the economics are usually straightforward: estimate per-request savings from your own test, multiply by expected request volume, and compare that with engineering, retraining, evaluation, and ongoing maintenance costs. Don't treat the reported 77% as a forecast - treat it as a reason to run a controlled experiment.
The decision rule is simple: test vocabulary specialization when repeated output and domain constraints show up clearly in production traces, and keep it only if the measured savings survive quality evaluation. Fewer tokens is an intermediate metric. The business outcome is lower cost per valid, useful response.