PhD student at UPenn
Joined January 2020
- Tweets25
- Following124
- Followers77
- Likes104
Pinned Tweet
So honored to have the support of my professors and peers, including the omnipotent Long @LongLeRobot, on my first PhD project. 3D space is the bridge between the task and action space, where the guidance from foundation knowledge flows.
VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored.
@TimSong52005757, @LongLeRobot, and our @GRASPlab team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time.
We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies.
This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence.
Project: omniguide.github.io/
Paper: arxiv.org/abs/2603.10052
TimSong retweeted
🧱 A little LEGO world, built with a coding agent!
Explore, take apart, and rebuild 3D LEGO sets right in your browser.
linzhanmou.com/resources/leg…
TimSong retweeted
Over the past few weeks, GPT-6 Astra, Claude Code with Opus 5.5, and other recent foundation models have surprised many of us in graphics, CAD, and robotics. They can now work directly with Blender, CAD kernels, game engines, physics simulators, and real robots very well. New examples appear almost daily, faster than the publication cycle can capture.
📝To keep track, we maintain a Living Survey of frontier AI for Design & 3D Modeling & Robotics:
🌐 mit-cdfg.github.io/Survey-AI…
Currently, it organizes 243 cases from 345+ public showcases and developer reports across 3D modeling, CAD and industrial design, animation, simulation, and robot control, each linked to its original source.
(If you are working on related projects, it's better to check this out: )
Different from a traditional survey, three ideas guide it:
1. Horizon scanning beyond publication lag
In an era where frontier foundation model capabilities evolve rapidly, traditional academic publishing cycles could lag behind public community developments and empirical findings. This platform establishes a centralized, high-velocity empirical synthesis repository to provide researchers and engineers with timely visibility into ongoing developments across 3D generation, parametric CAD, and embodied robotics—anchoring these observations in objective evaluations of strategic opportunities and critical safety boundaries.
2. Demonstrations as a distributed record of use
We analyze the corpus of over 345 publicly documented showcases and developer reports as an extensive, distributed "crowdsourced user study." This framing captures how models operate when prompted across diverse geometry kernels (CGM, Open CASCADE), DCC software (Blender), physics simulators (Isaac Sim, MuJoCo, Genesis), and physical robot hardware—revealing real-world workflow friction, prompt overhead, and boundary failures that static benchmarks miss.
3. Ranking by open verifiability
A central challenge is balancing the timely collection of rapidly emerging results with the need for high-quality, evidence-based evaluation. Our approach is to rank cases by open verifiability—the amount of evidence that others can directly inspect.
Equally impressive demos can carry very different evidence, so every case is ranked by what others can inspect:
- Rank 1 · Demo + Implementation Code: public code, scripts or CAD/robot harnesses (availability/verifiability)
- Rank 2 · Demo + Interactive Web Link: a live web app, 3D viewer or cloud CAD project
- Rank 3 · Demonstration Media Only: video or screenshots only
Note: Rank 1 means the evidence is inspectable, not necessarily reproducible from a clean setup.
Gallery: mit-cdfg.github.io/Survey-AI…
Contributions are very welcome via pull requests on GitHub. We especially welcome Rank 1 contributions (demos with code) and Rank 2 contributions (demos with interactive webpages).
Many thanks to our collaborators: Jamison Meindl(@meindljamie), Akihisa Watanabe(@Akihisa_Wat ), Anna Deng, Tianyu Huang(@tyanyuy3125), Igor Sadalski(@igorsadalski ), Harrison Liang, Minghao Guo(@GuoMh14 ), Benjamin Tod Jones, Wojciech Matusik(@wojmatusik ).
TimSong retweeted
🪄 Bring your 3D characters to life with words!
Introducing UniMate: one unified model for text-driven animation across diverse skeletons—from humans and animals to articulated objects.
🎬 A rigged asset + a text prompt → motion.
No per-skeleton retraining needed!
🌐 linzhanmou.com/unimate/
TimSong retweeted
With the advent of Astra as a policy, we wanted to test it on a difficult dexterous task: tying a shoelace knot.
Given just one video of the human doing the task, Astra figured out how to perform the task on real YAM robots in our custom coding harness. It directly generates end-effector poses and gripper commands, using camera feedback to adjust its actions.
This is an early glimpse into a new era of in-context learning for robotics, where new skills can be acquired at test time.
Code and more demos coming soon!
TimSong retweeted
How can we improve robot generalization beyond collecting more demonstrations?
A robot can use inference to work out how to solve tasks it was never explicitly trained to perform. With learned models of the world, we can plan our future actions and goals before acting.
I wrote a research perspective, Generalization by Construction, on how learning can be combined with inference for flexible generalization.
yilundu.github.io/perspectiv…
Our work is accepted by CoRL2026🥳
The code is releasing, feel free to try OmniGuide in sim or real!
See you in Austin🎉
VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored.
@TimSong52005757, @LongLeRobot, and our @GRASPlab team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time.
We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies.
This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence.
Project: omniguide.github.io/
Paper: arxiv.org/abs/2603.10052
TimSong retweeted
🌟Your static 3D world models are now alive and interactable!
🚀Introducing NeuROK, a neural simulation framework that turns any static 3D object into an interactive 4D asset — no per-category physics, no physical annotations for training.
📄 chen-geng.com/neurok
🧵 1/n
TimSong retweeted
Excited to share that our work NeuralActuator: Neural Actuation Modeling for Robot Dynamics and External Force Perception has been accepted to #RSS2026!
Your robot — even a low-cost one — can feel external forces without torque or tactile sensors.
TL;DR: NeuralActuator is a neural actuator model that jointly predicts 1️⃣torque to capture the nonlinear and time-varying current–to–torque relationship of low-cost servos, 2️⃣external contact forces (and force detection gates) for sensorless force perception, 3️⃣and motor conditions that indicate each motor’s operating regime.
Here is a fast-forward video clip ⬇️ We are also covering more robots like LeRobot-S101 and Franka Panda.
More details coming soon.
TimSong retweeted
I’m so tired of writing rebuttals to this kind of “lack of novelty” review: “This paper trivially combines A, B, and C, so the algorithmic novelty is limited.”
Technically, most (if not all) robotics papers are convex combinations of existing ideas.
I still deeply appreciate A+B+C papers—especially when they deliver:
- New capabilities: the “trivial combination” unlocks behaviors we simply couldn’t achieve before
- Sensible & organic design: A+B+C is clearly the right composition—not some arbitrary A′+B+C′
- Nontrivial interactions: careful analysis of the dynamics, coupling, or failure modes between A, B, C
- Rehabilitating old ideas: A was dismissed for years, but paired with modern B/C, it suddenly works—and teaches us why
- System-level & "interface" insight: the contribution is not any single piece, but how the pieces talk to each other
- Scaling laws or regimes: identifying when/why A+B+C works (and when it doesn’t)
- Engineering clarity: making something actually work robustly in the real world is not “trivial”
- New problem formulations: sometimes the real novelty is in the reformulation—only under this view does A+B+C make sense.
Maybe worth keeping these in mind when reviewing the next A+B+C paper : )
TimSong retweeted
Really exciting work! This is exactly promising for test-time controllability. Agents shall perceive to guide and correct behavior actively.
VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored.
@TimSong52005757, @LongLeRobot, and our @GRASPlab team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time.
We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies.
This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence.
Project: omniguide.github.io/
Paper: arxiv.org/abs/2603.10052
TimSong retweeted
Excited about this project: we showed how, for challenging tasks, a VLA can "get by with a little help from its friends"😉 --- powerful perception models that infer geometry and more, by constructing 3D guidance fields for diffusion and flow policies.@LongLeRobot @TimSong52005757
VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored.
@TimSong52005757, @LongLeRobot, and our @GRASPlab team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time.
We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies.
This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence.
Project: omniguide.github.io/
Paper: arxiv.org/abs/2603.10052
TimSong retweeted
It was a lot of fun to see this in person. We see so much online, but theres nothing nearly as convincing as just doing a demo, first try, right in front of someone.
And its cool work too, addressing a serious shortcoming of current policies
For this project, we’ve also optimized the speed significantly and shown real-time demos to several visitors coming to Penn including @chris_j_paxton and @DJiafei. Here’s a video of the robot finishing a task in 20 secs autonomously
TimSong retweeted
For this project, we’ve also optimized the speed significantly and shown real-time demos to several visitors coming to Penn including @chris_j_paxton and @DJiafei. Here’s a video of the robot finishing a task in 20 secs autonomously
VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored.
@TimSong52005757, @LongLeRobot, and our @GRASPlab team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time.
We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies.
This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence.
Project: omniguide.github.io/
Paper: arxiv.org/abs/2603.10052
TimSong retweeted
Scaling VLAs with more robot data is just not enough.
OmniGuide shows you can fix generalist policies at inference time.
Add guidance fields in 3D space that attract toward goals and repel from obstacles, and steer the policy without retraining.
TimSong retweeted
Check out my most recent project that I've been working on for more than half a year -- co-led with Tim @TimSong52005757!
We showed a flexible and effective recipe for improving generalist VLAs leveraging non-robotic foundation models! The key is guidance in 3D!
More details @ omniguide.github.io/
VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored.
@TimSong52005757, @LongLeRobot, and our @GRASPlab team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time.
We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies.
This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence.
Project: omniguide.github.io/
Paper: arxiv.org/abs/2603.10052
TimSong retweeted
VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored.
@TimSong52005757, @LongLeRobot, and our @GRASPlab team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time.
We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies.
This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence.
Project: omniguide.github.io/
Paper: arxiv.org/abs/2603.10052
TimSong retweeted
Capture a scene with 2DGS that can be viewed from extremely close and far away viewpoints, without aliasing while maintaining detailed texture. Check our new work timsong412.github.io/HDGS-Pr… lead by @TimSong52005757 @CrossEntropi and advised by me, @LingjieLiu1 and @KostasPenn
