🤔 Have @OpenAI o3, Gemini 2.5, Claude 3.7 formed an internal world model to understand the physical world, or just align pixels with words? We introduce WM-ABench, the first systematic evaluation of VLMs as world models. Using a cognitively-inspired framework, we test 15 SOTA models (including o3 & Gemini-2.5-Pro) across 23 dimensions of perception and prediction. Key findings 🔍: 1. VLMs excel at basic visual recognition but struggle with 3D spatial reasoning, temporal & motion dynamics, and mechanistic simulation. 2. VLMs tend to entangle irrelevant physical concepts. For instance, color changes bias performance across most perception tasks. 3. On complex next-state prediction, even the best VLM shows a 34.3% gap compared to humans, revealing fundamental limitations in transitive and compositional inference. 4. Accurate perception does not guarantee accurate prediction. Even with relatively accurate state perception, VLMs lack the foundational physical knowledge to simulate object interactions correctly. 🧵 Thread with specs, results & implications below ⬇️ #VLM #WorldModel #LLM #ACL2025