NVIDIA Cosmos 3: Open Models for World Simulation
See what NVIDIA Cosmos 3’s four open-weight models can do, how they support interactive simulation, and what teams should consider before adopting them.
NVIDIA Cosmos 3: Four Models for More Interactive Simulation
Training a robot or autonomous vehicle in the real world is expensive, slow, and hard to scale. Simulators help, but the scenes you render don't always match what your sensors actually record. That mismatch matters: a policy that looks great in a "clean" virtual setup can fall apart when lighting changes, the camera shifts, or something in the scene behaves differently than expected.
NVIDIA's new Cosmos 3 checkpoints try to close that gap by generating observations in ways that line up with robotics and driving workflows. The release includes four open-weight models that can produce driving scenes (with different sensor outputs), synchronized camera and LiDAR observations, simulator-based video, or visual predictions from robot actions. Each checkpoint is a post-trained version of Cosmos 3 Nano with 16 billion trainable parameters, and NVIDIA also provides adaptation recipes for working with your own data.
The real question isn't whether the generated video looks convincing. It's whether the model outputs observations you can trust for training and evaluation - especially outside a demo setting. Below, I'll walk through what each checkpoint does, how streaming generation works, where these models might fit in a business workflow, and what you should test before you bet on them.
What Cosmos 3 adds to world simulation
Cosmos 3 is a world-model development platform. In practice, it's built to generate observations that change as the scene changes. This release emphasizes interactive simulation for robotics and driving, instead of treating video generation as the end goal.
That difference affects how you should evaluate results. A standalone clip can look coherent and still be useless for training. A simulation workflow needs a link between what you feed the model (state or action) and what it returns (the observation). For driving, that can mean changes in scene structure, traffic motion, weather, or signal states. For robotics, it means generating what a camera might see after a robot moves or manipulates something.
The four checkpoints give you different ways to "control" what gets generated. Cosmos3-Nano-Transfer-Auto generates multi-view driving data, including camera outputs and LiDAR outputs. Cosmos3-Nano-Sim-Auto turns driving controls into synchronized sensor observations over time. Cosmos3-Nano-Sim-Transfer uses a physics simulator's controls to generate RGB video. Cosmos3-Nano-Sim-Bimanual predicts visual observations in response to bimanual robot (or camera) actions.
All four models use 16 billion trainable parameters. NVIDIA says they use the OpenMDW1.1 license, which permits both commercial and non-commercial use under its terms. Open weights can make inspection and adaptation easier, but you still need to check the license, estimate compute costs, confirm data rights, and run validation in your own environment.
The upside is coverage. If certain conditions are hard or expensive to capture repeatedly in the physical world, you can generate or adapt synthetic observations for them. But coverage doesn't automatically mean representativeness, and representativeness doesn't automatically guarantee safe policy transfer. Synthetic data helps you expand the test space - it doesn't replace real-world validation.
Which Cosmos 3 model fits the job
If you want the simplest way to choose, start with the system you already have: recorded driving data, an interactive driving simulator, a physics engine, or a robot action stream. Each checkpoint expects different inputs and produces different kinds of observations.
Transfer-Auto: driving scenarios to multi-sensor data
Cosmos3-Nano-Transfer-Auto is built for offline synthetic driving-data generation. It can generate synchronized video across up to 11 camera views (including fisheye cameras) and supports joint RGB and LiDAR generation.
Per-camera World Scenario Map controls describe scene structure and traffic motion, while text prompts shape the visual appearance. You can also provide optional initial RGB images to anchor the look of a scene. For LiDAR, it uses aligned HD-map range-map controls.
This model is a good fit for teams that want broader driving-data coverage across things like lighting, weather, and time of day, while still keeping explicit structure for the scene and traffic.
It also makes sense as a starting point if you're trying to adapt Cosmos 3 to a particular sensor rig or driving domain. NVIDIA's stated approach is to post-train Transfer-Auto on local data, then use it for offline generation, or continue with causal training and distillation to create a streaming Sim-Auto variant.
Sim-Auto: driving controls to streaming observations
Cosmos3-Nano-Sim-Auto is the causal, distilled streaming version of Transfer-Auto. It generates short successive chunks of synchronized camera video and LiDAR by using generated history plus updated driving controls. Instead of producing a full clip in one go, this setup supports longer interactive rollouts.
You'll need more than the checkpoint to run this closed loop. A simulation runtime and a WSM renderer need to update controls based on driving inputs. Teams also need consistent camera captions and map controls; LiDAR generation requires aligned HD-map range controls. For a new sensor setup or domain, NVIDIA's recommended path is to adapt Transfer-Auto first, then distill that into Sim-Auto.
Sim-Transfer: simulator controls to RGB video
Cosmos3-Nano-Sim-Transfer sits between a physics simulator and a vision-based policy. The simulator provides motion, contacts, and physical state. The model takes spatial controls - such as depth, segmentation, edges, or blur - plus text prompts, then generates color observations.
The split here is intentional: the simulator stays the source of physical state, while Sim-Transfer renders what an observation could look like given that state. A typical loop looks like: action → simulator update → spatial controls → generated RGB → policy next action.
This can be useful if your team wants visual input from generated frames without asking a video model to infer the underlying physics.
Sim-Bimanual: robot actions to video
Cosmos3-Nano-Sim-Bimanual predicts how the environment changes in response to actions from a two-arm robot. It supports YAM and AgiBot embodiments, with control coming either from a policy or from human-operated joysticks. A single sequence can include both viewpoint changes and manipulation.
The action format is a core part of the model contract, not something you can casually "figure out later." The example mixed-action setup uses 59 control values, padded to 64 during training, with conventions for coordinate frames, rotations, gripper calibration, validity masks, and control domains. Timing matters too: T transitions require T+1 observations. Released inference actions are already normalized, while training data needs to be transformed into the model's action representation.
How interactive streaming generation works
Streaming generation produces output in pieces. The model takes a prompt, available controls, and generated history to produce the next short segment. It decodes that segment into images, updates its history, and continues once the next controls arrive. Smaller chunks let you influence output more often - but they also affect latency and how heavy the compute load is.
Cosmos 3 uses a Mixture-of-Transformers backbone, with separate text-processing and visual-generation networks connected through attention. The visual generator works from compressed video representations called latents. After the initial image, each latent time step corresponds to four RGB frames. Transfer predicts one step per chunk, while the optimized Bimanual setup in the technical report predicts two.
The model also denoises latent representations. Temporal-causal attention lets it use available history and controls while blocking future target observations. A key/value cache keeps attention information for the next prediction, so it doesn't have to recompute everything from scratch each step.
Training pairs that architecture with a four-stage recipe:
- Bidirectional adaptation: learn the domain and relationships between controls and video from complete recorded clips.
- Teacher forcing: predict chunks from earlier recorded observations and available controls, using clean answers and masking future targets.
- Consistency distillation: train a student with a frozen teacher so the student can generate each chunk in four denoising steps.
- Self-forcing: train using the student's own generated history, with a frozen teacher and an auxiliary score model to provide a distribution-matching signal.
Self-forcing addresses a real failure mode: a small prediction error can become part of the context for later predictions, and then drift can compound over time. Training with generated history exposes the model to that problem. It doesn't guarantee there will be no drift at deployment time, but it makes the model handle the issue as part of training rather than hoping deployment magically stays perfect.
How to adapt and evaluate the models
NVIDIA's published post-training recipes start from Cosmos 3 Nano (or compatible base weights) and cover the four stages. For a team, you should treat adaptation like an engineering project: validate data, manage checkpoints carefully, and use held-out evaluation. Don't treat it like a single "run fine-tuning and forget it" command.
A sensible sequence is:
- Choose the closest starting checkpoint. Transfer-Auto for driving data and sensor adaptation; Sim-Auto for streaming driving simulation; Sim-Transfer for simulator-driven visual rendering; Sim-Bimanual for an embodiment plus a robot action format.
- Validate the data contract before training. Confirm sensor alignment, timestamps, camera layout, map controls, action units, and coordinate conventions. For bimanual data, check the schema and make sure transitions and observations line up.
- Run a small end-to-end test. Verify environment checks, prepare a sample, and test both training and restart behavior before committing to a longer run. A short run confirms the pipeline works - not that the model has converged or that outputs are useful.
- Keep resumable checkpoints. Full training checkpoints preserve optimizer and other training state. Exported inference weights don't include that state, so they can't replace a resumable training checkpoint.
- Evaluate on held-out episodes. Keep scenes, routes, or manipulation episodes out of adaptation. Compare generated observations against the intended controls, and check whether downstream policies change under useful variations.
The release's stated Sim-Bimanual setup requires Linux, an NVIDIA GPU, Docker, and NVIDIA Container Toolkit. NVIDIA also reports testing the vLLM-Omni setup on one GB200. That's a test configuration, not a minimum-memory guarantee for other hardware.
Evaluation should go beyond watching videos. For driving, check synchronization across camera views and whether RGB and LiDAR outputs stay consistent with each other. For simulator rendering, verify that the generated image reflects the simulator's updated state. For robot interaction, test action-response consistency, viewpoint changes, and behavior over longer sequences. In every case, track failure modes and policy outcomes - not just sample quality.
Business value and the limits teams should plan for
The commercial case is real, but conditional. Generating variations can reduce how much you need to physically capture every scenario, and it can help teams explore rare or expensive-to-stage conditions. For driving, that might mean generating variations in weather, lighting, or traffic configuration. For robotics, it might mean testing action sequences in a visual environment before running repeated hardware trials.
But the savings aren't automatic. You still need data pipelines, enough GPU capacity, simulator integration, evaluation infrastructure, and people who understand the domain. If generated data doesn't improve held-out policy performance - or if it doesn't surface useful failures - the cost of running the model can outweigh the benefit of extra synthetic coverage.
There's also a line between appearance and ground truth. Sim-Transfer depends on the physics simulator for motion, contacts, and physical state; it generates RGB frames from those controls. A plausible-looking frame isn't proof that the physical interaction is correct. And if you're using synchronized sensor outputs, you still need to test cross-modal consistency rather than assuming the joint generation makes it reliable.
For safety-sensitive applications, generated scenarios should be treated as one layer in a broader validation system. Keep clear provenance for synthetic and captured data, record the controls used for each sample, and test policies against real sensor data and independent simulator cases. Review distribution shifts and failure modes - especially situations where a visually plausible output hides a meaningful mismatch.
The open-weight release and the post-training recipes give researchers and developers a shared starting point. That can make adaptation and comparison easier. Still, openness isn't the same thing as robustness. The core business decision is whether your team can measure the gap between the generated world and the environment where your system will operate - and whether closing that gap is worth the operational cost.
Key takeaways
- Pick a checkpoint based on the inputs and outputs your workflow actually needs, not on the broad label "world model."
- Transfer-Auto targets offline multi-view driving data; Sim-Auto adds streaming driving observations; Sim-Transfer renders RGB from simulator controls; Sim-Bimanual predicts video from robot actions.
- Interactive generation depends on synchronized controls, a valid data schema, runtime integration, and generated history.
- A training run tells you the pipeline runs. Held-out evaluation is what tells you whether the outputs are useful.
- Treat synthetic data as a supplement to real-world testing, not a replacement. Validate sensor consistency, policy performance, and failure behavior.
Conclusion
Cosmos 3 includes four checkpoints that target different simulation tasks, from multi-view driving data to robot-action video prediction. Whether they deliver value depends more on how well they fit your sensors, controls, simulator setup, and evaluation process than on whether the samples look good.
Start with the workflow you actually need, validate the data contract, and test against held-out scenarios before scaling up. If generated observations improve measurable policy performance or reveal failures you couldn't test efficiently otherwise, the extra model and infrastructure may be worth it. If not, more generated video just means more output to manage.