We beat LeWorldModel where it actually matters.
TL;DR: SIGReg + MSE is misspecified for learning dynamics. LeWorldModel ends up rewarding the model for forgetting the hardest-to-predict parts of a scene, which are exactly the dynamics we care about. That is backwards.
Temporal Discriminative World Modeling (TDWM) is a clean fix. It forces the model to discriminate dynamics rather than static appearance and outperforms LeWorldModel on planning.
As a result, the model no longer collapses, and it reaches a low SIGReg loss on its own without any direct supervision. Check out our blog for more details:
td-worldmodel.github.io/TD-W…
Thanks to
@williamgpeng for a great collaboration!