World models are usually judged by how well they predict what happens next. But a model can predict accurately and still fail at the thing you actually care about: choosing a sequence of actions that works. That gap matters if you want an AI system to navigate a task, not just describe a plausible future.
A set of 600 experiments on LeWorldModel (LeWM), a JEPA world model tested in the Push-T robotics simulation, suggests a practical lesson: prediction quality wasn't the main bottleneck. Without changing LeWM's weights, new planning methods pushed success on long tasks from 22% to 85% in one test setup. Other tweaks helped when demonstrations were sparse and when goals didn't spell out exactly where the robot should place its arm.
The results are useful because they separate "model quality" from "planner design." They also point to a limitation in the benchmark itself: the robot does well when goals look like what it has seen during training, and it struggles when asked to reach unfamiliar states. Here's what the experiments show - what improved things, what didn't, and what still needs work before these ideas transfer to real robots.
How does a JEPA world model plan?
A world model predicts how the environment changes after an agent takes an action. A common "generative" approach tries to predict the future by producing a full image (or some similarly detailed output). JEPA - joint-embedding predictive architecture - takes a different approach: it predicts a representation of the future, not a pixel-perfect reconstruction.
LeWM has three main parts:
- An encoder that turns an observation (for example, a camera frame) into a compact embedding.
- A predictor that estimates the next embedding from the current embedding and an action.
- A planner that tests possible action sequences and picks one where the predicted endpoint is close to the goal embedding.
The key idea isn't that the planner needs to get every visual detail right. It needs to model whatever parts of the scene actually affect decisions. In Push-T, the robot's job is to push a T-shaped block across a table until it reaches a target position. The simulation also lets researchers compare the learned predictor against the environment's actual physics.
That comparison led to a surprising result. On 250 long-horizon tasks - goals 75 steps away, with expert-provided subgoals - LeWM succeeded 78% of the time. Swapping in the simulator's perfect physics as the predictor produced 77% success. In other words: along those trained paths, improving the dynamics model didn't translate into better planning performance.
This doesn't mean the learned predictor is "always correct." It just suggests that, in this setup and for these paths, its errors weren't the biggest constraint. The bigger leverage came from how the planner used predictions - and from how the task itself was represented.
This distinction matters outside robotics too. When an AI system underperforms, it's tempting to assume the model needs more data or more capacity. Sometimes the real issue sits elsewhere: the search strategy, the task decomposition, what the planner is shown during training, or even what counts as success. Replacing a component with an idealized version can help test that assumption before you sink time and compute into a larger model.
Why did chaining subgoals improve long-horizon planning?
When used as published, LeWM succeeded on only 22% of tasks with a goal 75 steps away. But on tasks with goals 25 steps away, it succeeded 89.7% of the time. The gap pointed to a planning horizon problem: short segments were reliable, but asking for one long plan in a single pass stretched what the system could handle.
One explanation is that small errors build up across time. That might be true, but the experiments also compared alternatives. They tested the effect of "better physics" against changes in how the task was structured. Replacing the predictor with exact physics added about 20 percentage points in the relevant comparison. Splitting the long task into 25-step segments added nearly 60 points.
This is hierarchical planning: a high-level component picks intermediate states (subgoals), and a low-level component figures out how to reach each one. Instead of demanding a 75-step solution all at once, the system solves a chain of shorter problems that fall within the range it already handles well.
With expert demonstrations providing those subgoals, the two planning changes raised success from 22% to 85% on the long-horizon test. That result was close to what you'd expect if the planner were given ideal subgoals straight from the expert demonstrations. It also shows why "just run more search" isn't always a substitute. Running the planner four times and taking the lowest-cost plan used four times the compute but added less than one percentage point of success. The extra runs often didn't diversify the outcomes - they tended to succeed or fail on the same tasks.
When this approach helps - and when it does not
Subgoal chaining works best when the low-level system is reliable over a known horizon and the task can be broken into reachable steps. It reduces the burden on both the model and the search process. But there's a harder question underneath: if there's no expert demonstration to hand you subgoals, where do those intermediate targets come from?
That's the boundary of the result. Demonstration-based subgoals show that LeWM can execute a useful sequence. They don't automatically show it can independently invent a good sequence for an unfamiliar task. In a real deployment, the high-level planner has to propose intermediate states that are both achievable and aligned with what the user wants.
Key takeaway: If a model works over short horizons but breaks on long ones, start by testing whether the task should be decomposed. More accurate predictions may help later - but in this setup, getting the planning structure right mattered more.
How can a memory graph and transformed demonstrations help?
Expert subgoals are useful in a controlled experiment. A deployed robot, though, won't have a demonstration tailored to every request. LeWM's training data can offer another resource: a memory of states the model has already seen.
In the experiment, researchers embedded roughly 460,000 frames from demonstrations and connected them in a graph. Each state linked to later states within its own demonstration and to nearby states drawn from other demonstrations. The planner then searched for a path through that graph - from the current state to the goal. The states along the path became subgoals for the low-level planner.
This setup depended on the embedding being tied to physical reachability. In a test with one reachable subgoal and one that couldn't be reached within 25 steps, embedding distance ranked the reachable option as closer 87% of the time. So the representation worked not only as a description of scenes, but also as a rough map of what the robot can reach.
The first graph design had a failure mode. States from different demonstrations could land close together in embedding space even when the robot couldn't move directly between them in physical space. Because cross-demonstration transitions were treated like valid moves during search, the planner could stitch together routes that didn't make sense physically. Adding a cost to each "jump" fixed much of the issue. On the same 250 long-horizon tasks, the memory-graph planner's success rose from 62% to 85%, matching the score achieved with expert subgoals.
Why memory coverage matters
A memory graph can only plan through states it contains (or through connections that it can justify as plausible). In an experiment where stored states near the goal were removed, success fell from 93% to 32%. Push-T offered a workaround: the physics stays the same if you shift, rotate, or mirror the whole scene. The system used that fact to transform demonstration segments from other regions and use them as candidate paths in the missing area.
With transformed segments, success recovered to 85% on that test. The approach also made a smaller dataset more useful: a graph built from 1% of demonstrations plus transformed copies performed about as well as one built from 10% of the demonstrations.
The benefit wasn't uniform. On a harder test requiring travel between starts and goals drawn from different demonstrations, transformed segments raised success from 45% to 56%. They helped most when the goal moved against the direction that demonstrations typically followed: in that case, success rose from 27% to 46%. Overall, the method improved coverage, but it didn't remove the need to reason beyond what the raw data covers.
Why does goal scoring change a planner's results?
A planner optimizes the objective it's given - including details that may not actually matter. In Push-T, the goal is represented by an image. That image includes both the block and the robot's pusher. If the planner compares its predicted scene to the entire image, it may try to match both objects: put the pusher where it appears in the goal and also move the block to the target.
That can create a plan that looks correct by the metric, but is wrong in practice. In one interactive test, the robot pushed the block to within 14 pixels of the target, then moved it away while trying to position the pusher where it appeared in the goal image. It wasn't ignoring the objective. The objective had accidentally included an unnecessary detail.
To isolate the effect, the experiment ran a planner using the simulator's exact physics on the same 50 harder tasks. The only change was whether the cost function included the pusher's position. Thirty tasks switched from failure to success when the pusher was excluded; none flipped the other way. On those "fixed" tasks, the planner that included the pusher put it within half a pixel of the pictured location - even though the block was no longer correctly held at the target.
LeWM can't just be told to ignore the pusher as a separate variable because its goal embedding combines objects without labeling them independently. Instead, the experiment compared each prediction against 16 versions of the goal image: the block stayed fixed in the required position, while the pusher position varied based on stored demonstrations. The planner selected the closest match. Without retraining, this change raised success on the harder test from 11% to 23%.
Specify acceptable outcomes, not a single picture
The broader design lesson is simple: goals should represent a set of valid outcomes, not one arbitrary image. "Put the cup on the shelf" doesn't usually require the robot arm to end in one exact pose. But an image-based objective can unintentionally force that kind of constraint.
In a real system, goal representations should state what must be true while allowing variation in what doesn't matter. The robot may need to place an object securely and end in a stable state, while the arm position can vary. Demonstration datasets can help here: they often include the same object arrangement with different robot poses, which helps define the acceptable range.
This isn't just a robotics issue. In any AI workflow, a poorly specified metric can reward proxy behavior instead of the user's actual outcome. Better model accuracy won't fix an objective that's asking for the wrong thing.
What do these experiments mean for AI development?
The experiments offer a practical evaluation recipe for model-based systems: separate the quality of the world model's predictions from the quality of the decisions built on top of those predictions. In this work, replacing LeWM's predictor with exact physics provided a controlled upper-bound test. Since that replacement didn't beat the learned predictor on the tested paths, the results shifted attention toward planning and goal design.
That's especially relevant when budgets are tight. Training can be expensive, and simply making models bigger doesn't automatically fix failures caused by bad task decomposition or an overly narrow success criterion. By testing planning and evaluation changes around a fixed, trained model, teams can learn where the next investment is likely to pay off - learning, planning, data coverage, or evaluation.
The benchmark still needs careful interpretation. The standard Push-T test uses a demonstration, starts partway through it, and sets a goal a fixed number of steps later. Because demonstrations end with the block in the same place, many goals already lie on routes represented in the memory. In the analysis reported, 84% of standard-test goals had a nearly identical state elsewhere in the demonstrations.
Episode count also mattered. LeWM succeeded on 68% of the official 50 episodes with a 50-step goal, but on 250 fresh episodes drawn the same way, success dropped to 44%. Changing only the planner's random seed changed outcomes on 38 of those 250 episodes. With a small test set, performance can look more stable than it really is.
The harder test addressed one limitation by using a start from one demonstration and a goal from another. It also required the block to remain on target for the final 10 steps. The memory-graph planner succeeded on 75% of standard tasks, but only 45% of the harder ones. Transformed memories raised the harder-test result to 56%. That's progress, but it's not evidence that the system can solve arbitrary goals.
What remains unsolved
The open problem here is proposing useful subgoals when demonstrations don't provide a direct route. These experiments show that LeWM can plan using demonstrated experience, reuse it through transformations, and avoid some misleading goal details. What they don't show is reliable "from-scratch" strategy invention for goals that conflict with the data.
That distinction matters when people interpret world-model benchmarks. Strong results can show a system plans well within the covered distribution. They don't, by themselves, establish robust generalization, safe behavior, or readiness for real physical deployment. The next step isn't just "scale the model." It's to test whether a higher-level planner can find valid routes beyond what the demonstrations cover and still perform under less convenient conditions.
Conclusion
The clearest lesson from these 600 experiments is that a world model's usefulness depends on more than prediction quality. In the tested paths, LeWM's learned dynamics performed about the same as exact simulator physics. Changes to planning and goal scoring produced bigger gains without retraining.
For teams building model-based AI, the practical approach is to diagnose the failure before scaling up: test the predictor against an ideal baseline, check whether the task goes beyond the planning horizon, review how much experience the system can reuse, and verify that the goal actually matches what users want. In this case, that process surfaced real improvements - and made the remaining bottleneck more specific: finding good subgoals beyond the demonstrations.
