Apertus 1.5: Open Multimodal AI Explained
Explore what Apertus 1.5 changes in open AI, from image and audio input to long context and reproducible evaluations. See what it means for your team.
Apertus 1.5: What Open Multimodal AI Changes for Builders
It's tempting to summarize a model release as a stack of benchmark numbers and a bullet list of new features. The tougher question is what happens when you try to build with it: can it handle the documents your users actually send? Can your team see how it was trained? And can you rerun the evaluations before you ship?
Apertus 1.5, from the Swiss AI Initiative, pushes beyond its first release. The 8B and 70B models add image and audio understanding, a context window up to 262,144 tokens, a reasoning mode, and improvements to instruction following and tool use. The project also publishes training details, data information, intermediate checkpoints, and structured evaluation records.
In practice, that combination matters more than any single headline. Multimodal input can reduce manual handling in document-heavy workflows, and openness gives teams more to inspect before they commit. But none of this removes the need for testing - spoken-language processing is still experimental, and the models lag some leading open-weight systems in certain areas. Below is what Apertus 1.5 actually offers, how the team developed it, and where it's likely to fit - along with where it probably won't.
What Is New in Apertus 1.5
Apertus 1.5 is a continued-pretraining release, not a ground-up rebuild. The team added 4 trillion tokens of text and multimodal training data to the 8B model and 2 trillion tokens to the 70B model. The release notes say the resulting models were trained on 19 trillion and 17 trillion tokens, respectively.
That's a useful distinction. This update builds on the first generation instead of replacing the whole foundation. The stated goal is to preserve and extend the original language capabilities while adding new input types and improving instruction following and tool use.
Images and audio enter the same sequence
The headline change is multimodal input. Apertus 1.5 can take text alongside images and audio, including practical inputs like photographs, diagrams, and documents - not just text pasted out of them. It also supports spoken-language processing, but the release calls it experimental, so you'll want to test it separately from text+image document workflows.
The project uses an early-fusion approach: images and audio are converted into discrete tokens and placed in the same sequence as text tokens. It does not introduce a separate vision encoder. Images are tokenized with the Emu3.5 tokenizer; the release gives 1,024 tokens for a 512-by-512 image. Audio is tokenized with WavTokenizer at a stated rate of 40 tokens per second.
For a product team, the boundary is straightforward: the models understand image and audio input, but they generate text only. Apertus 1.5 can answer questions about a chart or an audio clip, but the release isn't positioning it as an end-to-end system that outputs images or speech.
Longer context and a reasoning option
The context window grows to 262,144 tokens - four times the 65,536-token window of Apertus 1.0. A longer window can make it easier to include more of a report, codebase, or related documents in a single request. It doesn't guarantee the model will use every detail in a long prompt, though. Retrieval strategy, prompt design, latency, and evaluation setup still matter.
Apertus 1.5 also supports a thinking mode. The model can reason about an input before producing an answer, and the release reports improvements on reasoning-heavy tasks when this mode is used. Thinking should be treated as a configurable capability, not a magic switch. Compare accuracy, response time, and cost on your own tasks.
Other changes include more consistent instruction following and improved training for tool integration. Those can help models fit better into real workflows, but tool use only works reliably when the surrounding system handles permissions, validation, failures, and logging. A model that can call an API isn't automatically a safe or dependable agent.
How the Models Were Trained
Apertus 1.5 is worth looking at not just for its outputs, but for how the project says it got there. The release describes a staged process covering continued pretraining and post-training. If you're evaluating an open model, those details answer questions a benchmark number alone can't: what data was used, which training stages mattered, and what intermediate versions are available to inspect.
Four stages of continued pretraining
Starting from Apertus 1.0, the team continued pretraining in four stages:
- Unpaired data: Text and standalone images and audio introduce the new modalities.
- Paired data: Image-caption and audio-transcript examples connect visual and audio inputs with language.
- Cooldown: Training uses higher-quality data, with more emphasis on mathematics and code.
- Long-context extension: The context window increases progressively from 8,000 to 262,000 tokens.
More than half of the training tokens are text-only, which the team frames as a way to preserve the language abilities of the previous generation. They also refreshed text data with sources including FineWeb2-HQ+, FinePDFs, and Nemotron-CC. The release says the new data (including multimodal data) had to meet requirements for public availability and permissive licensing, and that robots.txt opt-outs were respected retroactively. Source information is laid out in the project's data transparency materials.
Those disclosures help, but they don't remove every decision you still have to make. A permissive license and a documented source list are useful inputs for legal and governance review; they don't replace checking what applies to your specific commercial use, jurisdiction, and datasets.
Five post-training stages
After pretraining, the models went through supervised fine-tuning, long-context fine-tuning, reinforcement learning with verifiable rewards (RLVR), direct preference optimization (DPO), and charter alignment through SDPO.
The supervised stage used 3.7 million text samples, along with image and speech data. The team says it added reasoning traces only for prompts the model could not already solve without them, and it limited tool-use examples to trajectories verified through execution. It also says it released the text SFT mix on Hugging Face.
Then came long-context supervised fine-tuning to restore capabilities at the full 262K-token window - important, because short-context fine-tuning can degrade performance on long inputs. RLVR used around 350,000 prompts spanning mathematics, code, instruction following, multilingual tasks, and vision. DPO used judge-scored preference pairs to improve chat quality, tool use, and safety. The final SDPO stage aligned the 8B model with the Apertus Charter through judge feedback and self-distillation.
The project also reports ablation results: mixing multimodal data during supervised fine-tuning helped across modalities, and multimodal reasoning traces continued to support mathematics performance even after text-only reinforcement learning. Those observations are more useful than a blanket statement like "more data helps." They give other teams hypotheses they can test against their own training setups.
For more detail, the release links to the Apertus 1.5 technical report, plus model collections, datasets, and code. Intermediate checkpoints include base, SFT, RLVR, and DPO variants, so researchers can compare changes across stages instead of treating the final model like a black box.
What the Evaluations Tell You
The project expanded its evaluations beyond the text benchmarks from Apertus 1.0. The new coverage includes image and audio tasks, thinking-mode evaluations, translation, multilingual performance, safety, and tool use. Multimodal evaluations use an MLLM-eval-suite pipeline that combines lmms-eval and VLMEvalKit; text evaluations build on a public fork of lm-evaluation-harness.
The results are mixed, in the way useful model assessments often are. The release says Apertus 1.5 improves coding and instruction following, and that the 8B model outperforms similarly sized open-weight models on instruction-following and multilingual tasks. Thinking mode improves performance on harder reasoning benchmarks like AIME. For multimodal tasks, the release describes results as competitive on areas such as spatial understanding and document or chart understanding, while also noting that Apertus remains behind frontier open-weight models overall.
That nuance matters. A strong result on one benchmark doesn't mean the model will be "best" across a broad product category. It tells you something about performance under a specific dataset, prompt style, metric, and inference setup. If your real job is extracting fields from scanned forms, for example, general visual question answering benchmarks are only a starting point - you still need to test your inputs and measure error rates.
Evaluation Cards make results easier to interpret
Apertus is working with the EvalEval Coalition to publish results in the Every Eval Ever (EEE) schema. Evaluation Cards aim to capture more than the score: they include details like who ran the test, how the model was accessed, the generation settings, and how the metric should be interpreted. Those specifics can explain why two leaderboard results that look comparable aren't necessarily comparable in practice.
This is a practical response to a real procurement problem. Teams often see a benchmark number referenced without enough information to reproduce it or understand the conditions behind it. Structured records make comparisons more useful for both researchers and product teams, and they can support governance work. The release claims the EEE schema can represent 56 of the 60 evaluation-reporting elements identified across EU AI Act Article 55 and the GPAI Code of Practice requirements it reviewed. That's about reporting coverage - not a guarantee that using the schema by itself satisfies legal obligations.
The release also points to use-case evaluations. It reports strong performance on Swiss legal tasks (on par with leading open-weight models) and says Apertus 1.5 8B performs competitively on the Last Translation Benchmark. Domain-specific results are worth digging into if your organization works in legal or translation contexts - but they still need to be checked against your languages, your document formats, your review process, and your tolerance for risk.
Where Apertus 1.5 May Fit in a Business
The real question isn't whether multimodal models are useful in general. It's whether this specific model - its capabilities and its limits - improves a defined workflow enough to justify integration and ongoing oversight.
Apertus 1.5 is a candidate worth evaluating when you want an open-weight model that can work with text and images, has a long context window, and lets you inspect published training and evaluation materials. Potential uses include:
- Document review: Ask questions about reports that include prose and charts, then compare answers with human-reviewed ground truth.
- Internal knowledge workflows: Use longer documents or related materials in one context window, and measure whether retrieval or chunking would be more efficient.
- Multilingual support: Test language-specific performance on real customer questions, especially if existing benchmarks suggest a fit.
- Developer tooling: Evaluate instruction following, code assistance, and constrained tool use with checks around generated changes and API calls.
- Audio-informed workflows: Explore spoken input, but treat it as experimental and validate transcription-sensitive tasks before rolling anything out to end users.
Think of these as evaluation targets, not guaranteed wins. The safest approach is to start with a narrow task and define what "good enough" means before choosing a model. For document workflows, "good enough" might include field-level accuracy, citation or evidence requirements, review time, and what share of cases still need human intervention. For tool-using workflows, it could be successful completion rate, invalid action rate, and how well the system recovers after API errors.
A practical evaluation sequence
If you're evaluating Apertus 1.5, you can reduce risk with a simple progression:
- Choose one workflow. Don't test a broad idea like "an AI assistant." Pick a task with a clear user, inputs, and expected outputs.
- Build a representative test set. Include routine cases, edge cases, low-quality inputs, and examples the system should decline or escalate.
- Compare the full workflow, not just the model. Test against your existing process and relevant alternatives. Include preprocessing, retrieval, human review, and tool execution where applicable.
- Set acceptance thresholds. Track task accuracy, latency, cost, review burden, and failure severity. Averages can hide the errors that actually hurt.
- Run a limited deployment. Log outcomes, add human escalation, and monitor drift before expanding access.
Open weights can help with hosting flexibility and adaptation, but they don't make running a system free. You still need compute, serving infrastructure, security controls, monitoring, and a plan for upgrades. A managed API can reduce operational overhead while limiting control over deployment. The right choice depends on team capacity and the consequences of failure - not just on the word "open."
Apertus 1.5 is also part of a broader market shift: models are increasingly evaluated not only for text generation, but for multimodal handling, tool use, and more inspectable evidence about how they were trained. For businesses, the useful signal isn't that every organization should self-host a multimodal model. It's that capability, transparency, and operational fit increasingly have to be assessed together.
Conclusion
Apertus 1.5 is a real evolution of an open model family: it brings image and audio understanding into the same token sequence as text, expands context to 262K tokens, adds a reasoning option, and publishes a broad set of training and evaluation materials.
The strongest reason to pay attention isn't that the release automatically makes open models "better" for every business. It's that teams can inspect more of the evidence, explore intermediate checkpoints, and test the model across text and multimodal workflows. The limits still matter: speech is experimental, benchmark results are task-dependent, and deploying open weights still takes operational capacity.
If Apertus 1.5 looks relevant to your work, start with the release notes and linked technical materials. Then pick a narrow use case, test it on your own data, and make the deployment decision based on measured results - not model size or release headlines.