Holger Molin (@holger_molin) profile picture
Joined September 2021
We beat LeWorldModel where it actually matters. TL;DR: SIGReg + MSE is misspecified for learning dynamics. LeWorldModel ends up rewarding the model for forgetting the hardest-to-predict parts of a scene, which are exactly the dynamics we care about. That is backwards. Temporal Discriminative World Modeling (TDWM) is a clean fix. It forces the model to discriminate dynamics rather than static appearance and outperforms LeWorldModel on planning. As a result, the model no longer collapses, and it reaches a low SIGReg loss on its own without any direct supervision. Check out our blog for more details: td-worldmodel.github.io/TD-W… Thanks to @williamgpeng for a great collaboration!
3
28