Robot Learning 🚀 ToyBlocks Breakthrough: Sim→Real in 2026
Discover how ToyBlocks builds a robot learning benchmark with simulation-to-real transfer—showing failures, improving robotics AI evaluation, and testing on real robots in 2026.
ToyBlocks Turns Robot Learning Into a Testable System
A robot that completes a task once isn't necessarily learning to do it reliably. It might be copying a familiar layout, benefiting from a carefully chosen demonstration, or getting lucky in a way that falls apart when you change the object position. In robotics, it's hard to untangle these possibilities - running the same physical experiment over and over takes time, equipment, and careful setup.
Ai2's ToyBlocks project tackles this with a controlled simulation built around a robot arm and tabletop blocks. The setup is intentionally narrow, but the tasks still have real bite: the blocks can be pushed, flipped, stacked, and inspected for letters or symbols, and the robot may need to uncover hidden information before it can act. Researchers can generate training demonstrations, vary what the robot experiences, and then test the policy on tasks withheld from training.
That's why ToyBlocks is more than a block-stacking demo. It's a framework for studying how robot policies generalize, how high-level planning interacts with low-level control, and where simulation results do - or don't - track what happens on real robots. Here's how the system works, what Ai2's initial experiments found, and what teams building (or buying) robotics AI can take from it.
What is ToyBlocks, and why does robot learning need it?
ToyBlocks is an open simulation framework from Ai2 for training and evaluating robots in a constrained tabletop setting. A simulated arm interacts with blocks whose faces vary, including letters and pictures. Tasks can include finding a character, turning a block to reveal it, and arranging multiple blocks into a target structure.
Keeping the domain small isn't just convenience - it's how researchers get clean evidence. In an open-ended environment, you have to account for too many object types, surfaces, and interactions at once. Then it's difficult to tell whether a model fails because it can't plan, because it misread something, or because the physics behaved differently than expected. With a tighter domain, you can vary conditions in a systematic way while still testing perception, reasoning, and dexterity.
The problem with demonstrations alone
Robot learning often starts with demonstrations: recorded examples of a robot completing a task, which then become training data. In the real world, collecting those recordings can require teleoperation. That's slow, expensive, and it can overrepresent "the situations you happened to collect," instead of the situations you actually care about.
A model trained on demonstrations may look great on scenes that resemble the training setup, but struggle when objects move, when a block gets rotated, or when the task requires a new sequence of actions. If you only report success rates from uncontrolled evaluation, you can't tell what the model learned or where it tends to break.
ToyBlocks is designed to make those questions easier to ask. Researchers can generate demonstrations automatically, adjust the mix of skills included in training, and evaluate on tasks held out from training. They can also vary block size, table height, and other physical properties between scenes, so robustness to changes in geometry becomes something you can measure instead of hoping for.
A small world with deliberately varied tasks
The environment includes more than basic picking and stacking. Block faces support search, sorting, and arrangement tasks, while hidden letters create a reason to turn objects and gather information. A model might need to locate letters to spell a word, figure out which block to inspect next, and then build the stack in the right orientation.
That gives you a useful distinction: it's narrow, but it's not simplistic. ToyBlocks limits the object domain, yet it still tests multi-step planning, perception, contact-rich manipulation, and generalization. The point isn't "can the robot handle everything," but "does a specific change to training or model design improve a clearly defined capability."
Ai2 describes ToyBlocks as its first FoundationWorld: a controlled simulated environment paired with task and demonstration generation tools, plus measurements for evaluating robot behavior. The bigger idea is to make research environments that other teams can compare against, instead of treating every robotics experiment like a one-off custom build.
How does ToyBlocks generate data and evaluate robots?
ToyBlocks connects skill learning, task planning, demonstration generation, and evaluation in one workflow. That linkage matters because a training pipeline only helps research if teams can trace how data is produced and then measure what changes when you adjust it.
From short skills to complete demonstrations
The system first generates short-horizon skills - basic behaviors like pushing a block, flipping it onto another face, or picking and placing it. Reinforcement learning and motion planning help create these behaviors in Isaac Lab, Nvidia's simulation framework, where the learned policies choose actions through trial and error.
Some skill generation can use "privileged" simulator information, like the exact position and orientation of a block. A physical robot can't access that directly, so it has to infer state from sensors. Still, using simulator ground truth during skill generation can make it practical to produce reliable example behaviors that would otherwise require human teleoperation.
A task-and-motion planner then selects and sequences the skills needed for a larger task. It plans the transitions between skills, executes the plan in simulation, and records the resulting demonstration. Each recording includes labels for the task or skill being performed. Ai2 reports that the framework can generate and execute hundreds of plans in parallel on one GPU, reaching up to 24 times real-time performance.
Data mixtures make hypotheses testable
The data generator lets researchers control what shows up in training. For example, a team can increase the proportion of rotation and pick-and-place demonstrations, then test whether those examples actually help a model learn to flip blocks. Researchers can also compare training from different workspace regions and evaluate in areas the model didn't see during training.
This turns vague questions - "does more data help?" - into sharper ones: which skills need more examples, how much data is enough, and whether a new training mixture improves performance on held-out cases beyond the scenes it resembles. In other words, the results can feed dataset design, not just generate bigger datasets.
ToyBlocks also evaluates more than "did it finish?" Alongside success rates, it reports execution speed, forces exerted on objects, and task progress, using details shared with the planning system. Those extra measures help separate "it finished eventually but in a sloppy or slow way" from "it finished efficiently and safely."
Separate planning errors from control errors
The framework supports hierarchical systems, where a high-level model picks a skill and a low-level policy executes it. For instance, a vision-language model might decide which block to flip, while a separate action model controls the arm.
Researchers can also use oracle components - reference planners or actions that provide a correct answer at one level - to pinpoint where failures come from. If the task fails with an oracle low-level controller but a learned high-level planner, the evidence points toward planning. If the planner is correct but the execution still fails, then the control policy is the stronger suspect. That's more actionable than staring at a single end-to-end failure number.
What does ToyBlocks reveal about reasoning and generalization?
Ai2's initial studies use the environment to look at low-level action models, high-level planning, tool-using agents, and action-conditioned world models. The theme is controlled evaluation: vary one kind of difficulty, keep the rest measurable, and focus on the failure mode instead of just the headline score.
Generalization is not one capability
A robot can see new object arrangements that still fall within its training range, or it can face positions and configurations outside that range. The first case is interpolation; the second is extrapolation. Handling one doesn't guarantee handling the other.
ToyBlocks tests action models on tasks like stacking, unstacking, flipping, and rotating. The experiments vary skill sequences, block counts, letters and colors, and the locations where objects appear. For example, a model might train on objects in one workspace region and get evaluated in another, or train on stack heights of two and five blocks and then be tested on heights of three and four.
Ai2 reports that popular pretrained action models showed weak generalization in its initial tests, even with large-scale pretraining. That doesn't mean those models can't generalize - it means broad pretraining doesn't replace the need for targeted evaluation. The controlled setting also gives researchers a way to compare fine-tuning approaches (including low-rank adaptation) and see which changes actually help on held-out cases.
Long-horizon tasks expose planning weaknesses
In a task like spelling a short word with blocks, the robot may need to reveal hidden letters before it can decide how to build the stack. It has to choose observations, remember what it found, select actions, and recover from mistakes during execution. Testing a single pick-and-place action wouldn't capture that full chain.
Ai2 evaluated a high-level vision-language model paired with a low-level manipulation model across tasks that ramp up in difficulty. Early on, the characters started upright and visible. Harder conditions included blocks tilted or with relevant characters hidden. The setup lets teams ask whether failures come from finding the right information, planning the sequence, or physically executing it.
Tool-using agents face a similar issue. In ToyBlocks, an agent can look at camera images and call skills like pick, flip, or rotate. In a search-and-spell experiment, half the scenes were missing at least one required letter - so the agent needed to recognize when the task couldn't be completed instead of just continuing to act. Ai2 reports that the agent either solved the task or correctly said it was infeasible in 63% of cases. Common errors included confusing similar-looking letters and using the wrong letter orientation.
The takeaway is more interesting because it's not a clean win story. The agent could plan and search, but perceptual mistakes and orientation errors still dragged performance down. Ai2 also found that agents can improve their tool descriptions by running experiments: descriptions written based on the agent's own tests achieved a 63% success rate, compared with 24% for default descriptions and 36% for descriptions written by a human expert. Those numbers are tied to the reported setup, but they illustrate how a controlled environment can test whether an agent is actually learning how its tools behave.
World models need contact-aware evaluation
Action-conditioned world models aim to predict how a scene changes after the robot acts. If these models are used to train or evaluate policies, predictions need to be accurate around contact - not just visually convincing over a few frames.
Ai2 compared video prediction, an agent-built physics simulator, and a latent world model using both successful and unsuccessful trajectories for skills like picking, flipping, and rotating. By balancing the examples, the team made it harder for a model to look "effective" just by predicting success all the time. They also found different strengths across models: one handled block movement and task outcomes better, another tracked the gripper but struggled with contact, and a third was faster and more accurate on short interactions while drifting over longer trajectories.
That short-vs-long distinction matters. A model can predict a single grasp well and still accumulate errors across a full task. In an additional experiment, adding 100 model-generated trajectories from the best video model increased policy success in an unseen region from 19% to 78%. That's a strong result in this benchmark - but it's not a general promise that synthetic data will always help every robot and every task.
How well do ToyBlocks policies transfer to physical robots?
Simulation is useful because it makes repetition cheap. It can also hide differences in friction, camera inputs, contact dynamics, and real robot behavior. A policy that succeeds only in simulation isn't ready to deploy. That's why ToyBlocks includes tests on a physical robot and compares simulation behavior to real-world outcomes.
Ai2 tested policies trained entirely on ToyBlocks simulation data, without real-world training examples. The experiments used a Franka FR3 arm and a Robotiq gripper across 30 scene configurations. Individual skills included picking, placing, flipping, and rotating. A longer task combined a high-level planner and a low-level action model to spell using three blocks.
For individual skills, the low-level model's success rate was similar in simulation and reality: 80.0% vs 81.1%. The hierarchical system succeeded in 55.5% of simulated episodes (111 of 200) and 50.0% of real-world episodes (15 of 30). Ai2 notes that more trials are needed to establish statistical confidence across tasks. The real-world sample is small, and that context matters - but the reported numbers are encouraging evidence of transfer, not a definitive claim about broad deployment reliability.
Transfer takes deliberate engineering
Ai2 used three steps to narrow the sim-to-real gap:
- System identification: Tune simulated robot properties, including joint friction and armature, to match the real robot's motion more closely.
- Domain randomization: Vary object size, friction, table height, lighting, backgrounds, materials, and camera parameters during data generation.
- Realistic inference inputs: Use camera observations at deployment instead of simulator-only state. For depth-based policies, the team generated depth maps from paired RGB images rather than relying on noisy depth-sensor readings.
These aren't interchangeable shortcuts. System identification helps match specific dynamics, randomization exposes the model to variability, and sensor-based inference checks whether the perception pipeline provides usable inputs. Even then, deployment teams still need to validate the relevant objects, workspace constraints, safety requirements, and failure recovery procedures in their own environment.
The business lesson is pretty direct: simulation can reduce the cost of experimentation, but it doesn't remove the need to validate. A controlled benchmark can help identify which policies deserve hardware testing and where they're likely to fail. It still can't certify safety or reliability in a warehouse, factory, or home by itself.
What should robotics teams take from ToyBlocks?
ToyBlocks is most useful as an example of how to structure robot learning research, not as a plug-and-play solution for every industrial need. Its value is in connecting controllable training data to explicit evaluations and then checking what holds up in the real world.
For a robotics team, a practical pattern looks like this:
- Define the capability, not just the demo. Spell out what the robot must do and which variations matter (unseen positions, hidden object features, longer action sequences).
- Split the pipeline into pieces you can test. Track perception, planning, and low-level control separately where you can. Otherwise, one component can hide another's weakness.
- Control the training mix. Make clear which skills and scene conditions appear in training, and test how adding or removing examples changes held-out performance.
- Measure what failure costs. Success rate helps, but it's not enough. Track task progress, time, contact forces, unsafe motion, and recovery behavior - especially when those factors affect the actual application.
- Use simulation to decide what to test on hardware. Transfer promising policies, compare failure patterns, and update the simulation or policy based on what doesn't match.
There's also an ethical and operational side. A robot failing isn't only a model-quality issue if it can damage goods, break equipment, or injure someone. Safe evaluation has to consider what happens when actions go wrong, whether humans are in the loop, and the limits of what you actually tested. ToyBlocks provides measurable signals like forces and collision-related outcomes, but teams still have to decide acceptable thresholds and validate them against real operating risks.
Ai2 announced an initial release that includes simulation assets, task and demonstration generators, generated data, trained models, training code, evaluation code, and validation sets. The reported initial dataset contains 85,336 long-horizon demonstrations, representing nearly 1,600 hours of robot execution data and 144 million frames. That scale is relevant to researchers studying data efficiency and benchmark design, and the narrow domain gives teams a starting point for building controlled tests of their own.
The limitations are just as clear. Blocks on a tabletop don't match the variety of real objects, clutter, people, and changing conditions found in many commercial environments. Results in this setting shouldn't be treated as proof that a robot can generalize to unrelated tasks. The next useful step isn't claiming the benchmark solves general robotics - it's applying the same experimental discipline to more complex environments and tracking what transfers.
Conclusion
ToyBlocks makes a strong case for why robot learning needs more than larger datasets and impressive-looking demonstrations. Teams need controlled ways to vary experience, hold tasks out of training, diagnose failure, and check whether behavior survives contact with real hardware.
The framework's block domain is intentionally limited, and the early results should be read with that scope - and the small real-world sample - in mind. Still, the core method is broadly useful: make training conditions explicit, measure capabilities separately, and treat physical validation as part of the research loop. For teams building robotics AI, that's a more reliable path than assuming a successful demo means the system has learned the task.