Sumaiya Rahman

A robot world model can show a grasp that never happened. In one example from the @WOB_WorldOracle paper, a real robot fails to grasp a piece of fruit. Given the same starting image and the same action sequence, an action-conditioned world model (ACWM) renders a clean, successful grasp. On screen, the task looks done. In the real world, it wasn't. @WOB_WorldOracle, from AGIBOT and NVIDIA with Princeton, UT Austin and the University of Toronto, is an evaluation platform for robot behavior models. Its central question is simple: given the same initial observation and the same low-level action trajectory, does the predicted future match what actually happens in the real world? That matters because an ACWM predicts how a scene evolves when a robot executes a sequence of actions, and that is not the same as predicting the next frame. The same motion can end in a successful lift, an empty grasp or a dropped object. The paper points to data generation, scalable policy evaluation and closed-loop policy learning as potential uses, and for all of them a plausible-looking rollout is not enough. It has to reproduce what physically happens. The paper calls this execution coherence and tests it with three stress tests: does the model obey the actions it was given, including bad ones; does that hold across embodiments and action sources; and does it survive long rollouts. Three design choices carry it. First, every prediction has a real twin. Existing embodied world-model benchmarks mostly condition on a text instruction or evaluate in simulation. Here the model receives the recorded initial observation and action trajectory, replays them open loop, and the rollout is compared with the paired real execution. The data covers four interaction types (rigid fruit, deformable towels, fluid pouring, articulated drawers), three embodiments (AGIBOT G2, YAM and a human hand), and a split where a policy drove a real robot in closed loop. That is 335 evaluation episodes in total, and the human-operated episodes run about two minutes each. Second, failure is part of the data on purpose. About 20% of segments in the AGIBOT training set are failed or suboptimal attempts, and about 40% in the human-operated evaluation set. A model that learned "every action ends in success" gets caught. Third, outcome and motion are scored separately, along four dimensions. Physical state correctness has a VLM check criteria written from the real keyframe, such as "the green cup is grasped by the gripper." Interaction spatial alignment matches DINOv3 features inside SAM3 masks of arms and objects against the real trajectory. Key interaction area fidelity scores only the interaction regions, combining PSNR, LPIPS and V-JEPA similarity by geometric mean, so one weak component sinks the score. Multi-view action coherence checks the robot arm across three synchronized cameras. The part I found most interesting is how the scoring itself was tested. Human raters agreed with the metrics in 88.1 to 93.2% of pairwise comparisons across 128 pairs. When the authors swapped in a wrong outcome keyframe, physical state correctness dropped 67.9% on average, against 34.3% for key-area fidelity, 20.0% for spatial alignment and 12.3% for multi-view coherence. Blurring only the background left the key-area score at 100, while blurring inside the masks lowered it step by step. Delaying one camera view moved only the multi-view score. Then the results across seven ACWMs, which the paper frames as performance patterns, not a verdict. Arm motion is far easier than object motion. On human-operated AGIBOT episodes, one model scored 78.8 on arm alignment and 29.1 on the object. Another scored 89.4 and 45.4. The paper reports the same gap for every model, embodiment and data distribution. Following the trajectory is also not the same as getting the outcome right. On the policy-rollout split, one model scored 62.2 on physical state correctness but 51.6 on spatial alignment. Another scored 57.0 and 78.7. Over full episodes, one model reached 96.96 on multi-view coherence and 60.26 on physical state correctness. Time hurts every model. Two models fell from 50.56 to 27.08 and from 53.99 to 26.60 between the 30-second and full-episode regimes, and one that led the rest through 60 seconds was overtaken at 70. Human hands were the hardest case: the strongest hand model reached only 15.2 on object fidelity. One honest note: the authors state that these comparisons are observational, not causal. The models differ in architecture, pretraining and action conditioning, and adaptation is not matched. AGIBOT has a dedicated training set, YAM relies on public data with a larger domain gap, and the human-hand models run zero-shot. Each trial also uses a single real execution as reference, so a physically plausible alternative outcome may be penalized. If a world model will judge policies or generate training data, hallucinated success is the worst failure mode, because it quietly inflates the score. That is why developers should bring their own models. Evaluating on @WOB_WorldOracle shows how a model performs on robot behavior tasks, reveals where its predictions diverge from what a real robot does, and gives a reference point for further research and improvement. If you are building models for robot behavior, bring your model to @WOB_WorldOracle and see how it performs. Platform: worldoracle-benchmark.github… Paper: arxiv.org/abs/2610.18826 Code: github.com/AgibotTech/WorldO…