Pinned Tweet
How do we tell a world model what’s interesting?
For JEPA, there are two open problems: what should be in the latent, and how do we put it there. We solved the second with spectral manifold learning. Spectral alignment is a framework for training world model embeddings to preserve a chosen similarity structure (i.e. depth, temporal distance, or other metadata).
Using our method, we get an average 51% relative improvement over LeWorldModel across three robot environments, including CALVIN. We also provide a theoretical analysis of the objective— check out the paper for details.
Paper: arxiv.org/abs/2610.15429
Blog: spectral-alignment.github.io…
Huge thanks to @holger_molin, @mar_baga, and @randall_balestr for a great collaboration! 🎉
The standard latent world-model objective has a problem: it rewards the model for forgetting the hardest-to-predict parts of a scene, which are exactly the dynamics you need for planning.
SIGReg + MSE is misspecified for learning dynamics. Temporal Discriminative World Modeling (TDWM) fixes it by forcing the model to discriminate dynamics rather than static appearance.
Not only that, it outperforms LeWorldModel on planning, doesn't collapse, and drives SIGReg loss down on its own with zero direct supervision on that objective. Check it out on our blog: td-worldmodel.github.io/TD-W…
Curious what @ylecun and others working on JEPA make of this. Had a great time working with Holger Molin @HolgerMolin on this project!
1
60


