Ph.D. student at MIT EECS, working with @phillip_isola and @vincesitzmann. Previous RA at @Oxford_VGG.
Joined August 2022
- Tweets87
- Following617
- Followers1.2K
- Likes304
Pinned Tweet
1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
In the past decade, deep learning models have come to possess unreasonably effective power through training. Yet the very heart of their making, the optimizer, is still mechanistic.
Our post is an attempt to breathe the magic that training has given us into the core of training itself, forming a loop that we hope will create more capability in the future!
8/9
Send our pip package (OPAL, OPtimizer with an Agent in the Loop) to your agent and ask it to equip your training with reactive optimizers!
💻 github.com/xingjian-bai/opal
Joint work with Jia-Hao Liang, @jayden_teoh_ and @vincesitzmann.
Was playing with Astra with @sizhe_lester_li when my dad sent me this photo from ~20 years ago.
Naturally, I asked Astra to rebuild the tank I’d made as a kid — cube by cube, stick by stick.
Childhood memories came rushing back : )
🤖 Made with AI
I gave GPT-6 Astra photos of things in my apartment to rebuild in Minecraft.
The little fella walked, jumped, and built scaffolding to place blocks in survival mode. No flying or direct world edits.
This was so fun to watch! More on this soon!
With @SimulatedAnneal
Xingjian Bai retweeted
"Video models are world simulators."
Ok — can they predict a ball bouncing off another ball, off another ball? 🤨
We find standard video diffusion breaks down as the chain of dependent events lengthens — and more compute isn't the fix.
Meet the seriality gap. 🧵
1/n
Video models can now generate everything from anime clips to guitar-playing cats; language models, everything from poems to programs. Why shouldn’t robot policies eventually inherit the same generality? In VERA, we explore one route through reduction: plan in video space, then faithfully translate the visual plan into embodiment-specific actions.
This was my first robot-learning project, and I couldn’t have asked for a better introduction. I’m especially grateful to @sizhe_lester_li for bringing me into the collaboration, to @vincesitzmann for his passionate support, and to the whole team for making it such a rewarding journey!
Robot learning is moving beyond policies built for one robot, one scene, one task.
At MIT, we’re exploring a different path: turning video world models into embodiment-agnostic robot policies.
Introducing VERA: a 14B video-to-action system that controls robots across embodiments, skills, and environments.
From zero-shot pick-and-place on a real Panda arm to contact-rich cube reorientation with a 16-DoF robotic hand.
Different robots. Different environments. Different tasks.
Same video planner. Same weights.
We’re open-sourcing everything so you can fine-tune VERA for your own robot setup too. Deep dive in the thread:
🔗 vera.csail.mit.edu
🧵 (1/7)
In our formulation, image tokenization and latent generation become two sides of the same coin. One model, one stage, from scratch—no pretrained encoders needed.
Especially excited about applying UNITE to modalities like molecules and crystals, where a pretrained DINO simply doesn't exist.
Very unforgettable collaboration with @ShivamDuggal4 and our amazing team at Adobe Research!
Tokenization & Generation power Large Models. But are they really separate?
Tokenization=Generation under strong observability
UNITE: An end-to-end training framework where one shared Generative Encoder (GE) performs both token. & latent denoising
Paper: arxiv.org/abs/2603.22283
Do causal video diffusers really need dense causal attention at every layer, every denoising step?
We looked inside and found: no. Causality is separable from denoising.
Here are two surprising observations that hold across architectures, training objectives, and scales.
Trained from scratch, SCD beat previous models in all metrics at 4x lower latency.
We also fine-tuned from WAN 2.1, matching the VBench performance of the best frame-wise autoregressive models, while having 35% lower latency than Self Forcing, >10x faster than original WAN 2.1.
Project page: xingjianbai.com/sparse-causa…
Code (train from scratch): github.com/xingjian-bai/spar…
This work would not be possible without my amazing collaborators, @guande_he , @xxunhuang, @zhengqi_li, @elishechtman, @zongze_wu. I truly learned a lot from you through this exciting journey.