Tan Lip-Bu Professor in AI @ NTU | Director, MMLab@NTU | Generative AI, world models & embodied AI
Singapore
Joined November 2010
- Tweets1K
- Following769
- Followers3.9K
- Likes2.4K
Pinned Tweet
๐ Introducing Mundo, led by my student @onelineluo: a customized world model that turns images of a place into an environment for exploration, editing, and simulation.
Explore new viewpoints, change objects and physical properties, or simulate robot navigation and manipulation โ all within the captured environment.
The key is an explicit, editable world state: code specifies actions, physics computes their consequences, and a video model renders the outcome using the input images as visual context.
Project: mundo-world-model.github.io/
Blog: huggingface.co/blog/Luo-Yihaโฆ
Code coming soon!
Meet MatAnyone-E: high-quality video matting, much faster. โ๏ธ
Compared with MatAnyone 2 at 4K:
โก ~7.4ร faster inference
๐พ Just 7.6% of the peak GPU memory
โจ Comparable matting quality
Runs at 73 FPS at 1080p and 29 FPS at 4K on a single NVIDIA H200.
Project: shuyun-wang.github.io/projecโฆ
Paper: arxiv.org/abs/2610.35724
Code (coming soon): github.com/Shuyun-Wang/MatAnโฆ
How does MatAnyone-E save GPU memory? By being selective about where to lookโand what to look up. ๐ง
Dense matching compares every current-frame feature with features stored from past frames. At 4K, those comparisons become expensive.
MatAnyone-E first takes a low-resolution look, then:
โข Refines only uncertain regions.
โข Searches only the most relevant memory locations.
โข Reuses the coarse results elsewhere.
In this demo, just 5.9% of query locations need refinement, using 15% of memory candidatesโskipping 99.1% of high-resolution matching pairs.
๐ฅ A long video. One coherent 3D scene.
Meet HiP3R: piecing together local 3D reconstructions to keep the scene consistent across thousands of frames. Watch the demo and explore the details ๐
Project: kkakkkka.github.io/HiP3R/
Paper: arxiv.org/abs/2610.31635
Code: github.com/HiP3Rs/HiP3R
Why is reconstructing a long video difficult?
Short sections can look good on their own, yet drift apart when stitched together. Revisiting the same place may produce misaligned geometry.
The challenge is making all these local reconstructions agree.
Has GPT-6-Astra Achieved Embodied Intelligence?
Across 12 benchmarks, access to training demonstrations and up to 10ร the execution budget push several tasks close to 100% success.
How does this translate to real robots? Watch our demos across 8 robot configurations to see its capabilities and remaining challenges.
Paper: huggingface.co/papers/2610.2โฆ
Project & demos: sensenova.ai/has-gpt-6-astraโฆ
GitHub: github.com/OpenSenseNova/Senโฆ
What if you could step inside a van Gogh painting?
We used Mundo to imagine new viewpoints from a single painting.
Take a look at this short demo.
Project: mundo-world-model.github.io/
Blog: huggingface.co/blog/Luo-Yihaโฆ
Chen Change (Cavan) Loy retweeted
1/6 Ropedia Super Week Day 2
How can we turn first-person human experience into structured 3D spatial intelligence?
For Day 2 of Ropedia Super Week, we're introducing two complementary research projects: HoliWorld and EgoXperience. Both explore how we can make first-person experience more useful for understanding the physical world.
Chen Change (Cavan) Loy retweeted
We developed PhysLDM, a high-quality neural simulator based on latent diffusion models (LDMs)!
Highlights:
- A spatiotemporal VAE that compresses dynamic 3D deformable meshes into a compact latent space.
- An LDM that models both global motion and local deformation.
- Fully differentiable, enabling challenging inverse problems!
We hope PhysLDM will make 3D simulation more accessible and ML-friendly!
Awesome work by Yu Zhang @yucrazing !
๐ Project: yucrazing.github.io/physldm
๐ Paper: arxiv.org/abs/2610.07609
Chen Change (Cavan) Loy retweeted
1/5 Ropedia Super Week Day 1
How can robots learn to predict the physical world before acting in it?
Today, we're kicking off Ropedia Super Week with Ropedia W0, a world model exploring how human experience can help robots learn to predict what will happen next.
Here's what we're building.
This motivates benchmarks grounded in human needs, without in-domain demonstrations, and with a clock that keeps ticking.
Technical report: arxiv.org/abs/2610.22471
Project page: sensenova.ai/has-gpt-6-astraโฆ
Code: github.com/OpenSenseNova/Senโฆ
Chen Change (Cavan) Loy retweeted
Ropedia Super Week | Oct 19โ23
6 breakthroughs. 5 days. A showcase of how first-person human data can advance embodied AI, from spatial reasoning to action and control.
Here's the lineup. โ
Mon: Ropedia W0, our latest robot world model.
Tue: HoliWorld + EgoXperience, our spatial data pipeline & full-stack data-to-model framework.
Wed: S-Agent 2.0, our evidence-driven spatial reasoning agent.
Thu: DONIC, our generalist controller for two dexterous hands.
Fri: Ropedia A0, our code-driven robot self-improvement framework.
Tomorrow, we'll take a closer look at our projects.
Follow Ropedia for a first look.
Chen Change (Cavan) Loy retweeted
DINOv2 has never seen a caption, and Qwen3 has never seen an image. We still aligned their embedding spaces without a single image-caption pair.
It even works when the images and the captions come from different datasets.
Project page: dominik-schnaus.github.io/unโฆ
โฌ๏ธ
Chen Change (Cavan) Loy retweeted
The surprising effectiveness of video diffusion models for hand motion reconstruction
Blog: huggingface.co/blog/Xingang-โฆ
Chen Change (Cavan) Loy retweeted
Our ancestors picked up a rock. They saw a hammer, a blade, and a way to survive. Humans didnโt conquer the world with bare hands -- we did it with tools. ๐ ๏ธ
Behind every tool we reach for is a tiny act of intelligence: "What do I need? What should I use? What do I do next?"
It's not just knowing what something IS. Itโs imagining what it could DO. Can current AI models reason the same way?
Meet #EgoTools , collab with @ropedia_ai.
๐ ropedia.github.io/egotools/
Chen Change (Cavan) Loy retweeted
๐ Excited to share our new work: VLANeXt Family: A Systematic Study of VLA Models from Core Recipes to Emerging Paradigms! @KangLiao929 @ccloy
Starting from 500+ controlled experiments, we systematically revisit the design space of Vision-Language-Action (VLA) models and distill 12 practical recipes for building strong VLAs.
Beyond the original VLANeXt, we further ask: Do these recipes still hold as VLA paradigms evolve? We extend VLANeXt along four emerging directions:
โข Model Scaling
โข Latent Action Models (LAMs)
โข JEPA-style Predictive Representation Learning
โข World Action Models (WAMs)
Together, these variants form the VLANeXt Family, providing a unified framework for studying both current and emerging VLA paradigms.
๐ Page: dravenalg.github.io/projectsโฆ
๐ Paper: arxiv.org/abs/2602.18532v5
๐ป Code: github.com/DravenALG/VLANeXt
Feedback and discussions are very welcome! ๐ค
Ever wished you could change the focus and aperture after taking a photo?
Our work AnyBokeh, led by my student @itsmag11 Xinyu Hou, explores this with physics-guided bokeh editing from a single image, even one that already contains defocus blur!
The key idea is to use the existing blur as a cue to the imageโs optical characteristics. This guides a generative editor to refocus the image and adjust depth of field, without first reconstructing an all-in-focus image.
Take a look at the examples: itsmag11.github.io/AnyBokeh/
Code available at: github.com/itsmag11/AnyBokeh
Most bokeh editors want a sharp photo first. We thought: ๐๐ต๐ ๐ป๐ผ๐ ๐๐๐ฒ ๐๐ต๐ฒ ๐ผ๐ฝ๐๐ถ๐ฐ๐ฎ๐น ๐ฐ๐น๐๐ฒ๐ ๐ฎ๐น๐ฟ๐ฒ๐ฎ๐ฑ๐ ๐ถ๐ป ๐๐ต๐ฒ ๐ฏ๐น๐๐ฟ? ๐
Meet AnyBokeh! Edit bokeh directly in blurred photos. No all-in-focus detour. No test-time calibration. โจ
#NeurIPS2026
Chen Change (Cavan) Loy retweeted
Most bokeh editors want a sharp photo first. We thought: ๐๐ต๐ ๐ป๐ผ๐ ๐๐๐ฒ ๐๐ต๐ฒ ๐ผ๐ฝ๐๐ถ๐ฐ๐ฎ๐น ๐ฐ๐น๐๐ฒ๐ ๐ฎ๐น๐ฟ๐ฒ๐ฎ๐ฑ๐ ๐ถ๐ป ๐๐ต๐ฒ ๐ฏ๐น๐๐ฟ? ๐
Meet AnyBokeh! Edit bokeh directly in blurred photos. No all-in-focus detour. No test-time calibration. โจ
#NeurIPS2026