MIT PhD student in Computer Science. @MIT @MIT_CSAIL @MITEECS robots, RL, embodied reasoning, video-language models
MIT
Joined November 2025
- Tweets130
- Following2K
- Followers2.2K
- Likes9.6K
Pinned Tweet
Excited to share SOLE-R1 is accepted to NeurIPS 2026!
General-purpose reward models are fundamental to unlocking real-world RL.
We show SOLE-R1's zero-shot reward prediction can serve as the SOLE signal for learning over 20 unseen manipulation tasks through online RL (i.e., randomly initialized policies, 0% starting success rate).
This work provides a strong proof of concept: RL with zero-shot dense reward prediction can do more than just increase success/speed for existing tasks; it can enable robots to learn entirely new tasks from scratch, without any demonstrations or ground-truth rewards, starting from zero competence.
Very proud of this project!
Webpage: philip-mit.github.io/sole-r1…
Thank you to my coauthors @thomas_weng, @_ericrosen, @BizaOndrej.
#NeurIPS2026
fine-tuning our reward models on @dm_robots task. Still early, but cool to see our model show near-perfect progress prediction on this unseen trajectory with interweaving errors and recoveries.
the task has a nice combination of factors that make it a good challenge for the reward model (and policy):
- bimanual + multi-view: requires coordinating both arms and reasoning across three camera views
- on-robot cameras only: no external cameras providing a global view of the scene
- precision: demands precise insertion of the actuator into the hole where the visual difference between near-miss and success is very subtle
- start state and goal state change every episode: different actuator to pick up, different hole to insert it into
the task is right on the edge of hard, but not too hard to get to >99% success rate
exciting to see @DominiqueCAPaul and @aurel_arnold make so much progress with this task in a very short period of time
Philip Schroeder retweeted
First time being on a podcast, and it's RoboPapers! 🤩
I had a great time discussing SOLE-R1 (philip-mit.github.io/sole-r1…) with @micoolcho, @DJiafei, and @philip_mit
Reinforcement learning is often seen as key to achieving scalable, highly reliable real-world robot performance, yet it requires good reward functions that can supervise the wide range of tasks that need to be performed. The obvious thing to try would be a large vision-language model (VLM), but these models often fail under common real-world conditions like partial observability and occlusion.
As a solution @Philip_MIT and @thomas_weng propose SOLE-R1, a video-language reasoning model trained to serve as a reward signal for online RL. Given real video, SOLE-R1 will reason over input video frames, generates reasoning traces, and predicts a reward signal. This allows online RL from random initialization on new manipulation tasks.
To learn more, watch Episode 111 of RoboPapers, with @micoolcho and @DJiafei!
the @RoboPapers episode on SOLE-R1 is out now! we break down our recipe for training a reward model that stays robust to reward hacking as the policy actively optimizes against its zero-shot reward prediction during online RL.
check it out if you're interested in real-world reinforcement learning or general-purpose reward reasoning models!
Reinforcement learning is often seen as key to achieving scalable, highly reliable real-world robot performance, yet it requires good reward functions that can supervise the wide range of tasks that need to be performed. The obvious thing to try would be a large vision-language model (VLM), but these models often fail under common real-world conditions like partial observability and occlusion.
As a solution @Philip_MIT and @thomas_weng propose SOLE-R1, a video-language reasoning model trained to serve as a reward signal for online RL. Given real video, SOLE-R1 will reason over input video frames, generates reasoning traces, and predicts a reward signal. This allows online RL from random initialization on new manipulation tasks.
To learn more, watch Episode 111 of RoboPapers, with @micoolcho and @DJiafei!
Philip Schroeder retweeted
Full episode dropping soon!
Geeking out with @Philip_MIT @thomas_weng on SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot RL philip-mit.github.io/sole-r1…
Co-hosted by @micoolcho @DJiafei
very fun time talking about how to train general-purpose reward models for real-world RL on @RoboPapers with @thomas_weng, @micoolcho, and @DJiafei
full episode coming soon!
Full episode dropping soon!
Geeking out with @Philip_MIT @thomas_weng on SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot RL philip-mit.github.io/sole-r1…
Co-hosted by @micoolcho @DJiafei
Philip Schroeder retweeted
Full episode dropping soon!
Geeking out with @Philip_MIT @thomas_weng on SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot RL philip-mit.github.io/sole-r1…
Co-hosted by @micoolcho @DJiafei
Philip Schroeder retweeted
Designing reward functions for real-world RL is a serious challenge in learning for manipulation.
Zero/few-shot reward models that “just work” is going to be a huge unlock for the entire field to go beyond behavior cloning!
Excited to share SOLE-R1 is accepted to NeurIPS 2026!
General-purpose reward models are fundamental to unlocking real-world RL.
We show SOLE-R1's zero-shot reward prediction can serve as the SOLE signal for learning over 20 unseen manipulation tasks through online RL (i.e., randomly initialized policies, 0% starting success rate).
This work provides a strong proof of concept: RL with zero-shot dense reward prediction can do more than just increase success/speed for existing tasks; it can enable robots to learn entirely new tasks from scratch, without any demonstrations or ground-truth rewards, starting from zero competence.
Very proud of this project!
Webpage: philip-mit.github.io/sole-r1…
Thank you to my coauthors @thomas_weng, @_ericrosen, @BizaOndrej.
#NeurIPS2026
Philip Schroeder retweeted
Currently looking at new research directions and SOLE-R1 keeps coming up!
Excited to share SOLE-R1 is accepted to NeurIPS 2026!
General-purpose reward models are fundamental to unlocking real-world RL.
We show SOLE-R1's zero-shot reward prediction can serve as the SOLE signal for learning over 20 unseen manipulation tasks through online RL (i.e., randomly initialized policies, 0% starting success rate).
This work provides a strong proof of concept: RL with zero-shot dense reward prediction can do more than just increase success/speed for existing tasks; it can enable robots to learn entirely new tasks from scratch, without any demonstrations or ground-truth rewards, starting from zero competence.
Very proud of this project!
Webpage: philip-mit.github.io/sole-r1…
Thank you to my coauthors @thomas_weng, @_ericrosen, @BizaOndrej.
#NeurIPS2026
Philip Schroeder retweeted
Working on first real-world RL token experiments today.
> I’m only activating it for the critical part (insertion) as in paper
> first rollouts with RLT work decent
> After ~10 rollouts with updates the policy gets very jerky - maybe because I’m injecting noise for exploration or the residual policy isn’t regularised strongly enough. Noise injection should show up as jerkiness is first rollouts as well though?
In the video you can see the jerkiness. Also, once I turn the RLT network off and unedited VLA takes over everything runs smooth and actuator is inserted cleanly.
Philip Schroeder retweeted
Two months of ablating π0.5 finetunes on a real manufacturing task. The policy now succeeds 98% of the time.
We're publishing all results, all the data, and every run, including the ones that went nowhere.
What we learned:
1/ 5x more data took us from 63% to 76%. Pure scaling was the weakest lever we found.
2/ Diversity got more out of the same hours. 4h spread across five scenes beat 4h in the eval scene by 30pp.
3/ Quality did even better. One clean, slower hour on top of the 21h: 76% → 90%. That single hour added more than the 17 hours before it.
4/ So we tried quality on its own: 1h of clean data plus 240 rollouts with human interventions. 1.7h in total, 28% → 88%. That's 12 points above the 21h model with a twelfth of the data.
5/ Inference settings add more on top, with no retraining. They took the 21h model from 76% to 93%, and the 21h+1h model from 90% to 98%.
Just as useful: what didn't matter. Learning rate, batch size, relative vs. absolute joints, image augmentation. All less important than we expected.
Full write-up linked below.
Philip Schroeder retweeted
Blog post out for our RL post-training method, Q2RL!
Looking to improve your deployed robot with reinforcement learning in the real world?
Presented at RSS 2026, Q-Estimation and Q-Gating from Behavior Cloning for Reinforcement Learning (Q2RL) improves robot policies in a few hours of real world interaction, avoiding catastrophic forgetting when fine-tuning a Behavior Cloning (BC) policy via Reinforcement Learning (RL) post-training.
Q2RL estimates a Q-function for the BC policy, and uses it to guide online RL. Learn more about the method here: rai-inst.com/resources/blog/…
Philip Schroeder retweeted
Exciting to be organizing a #CoRL2026 workshop on real-world reinforcement learning!
Post-training in the real world will be critical for deploying useful robots, but it's currently being slept on. The R2RL workshop brings together researchers working in this area to network and share ideas.
If you're pioneering this research, consider joining us at the workshop! Workshop submissions accepted until Sep. 30.
We’re trying something new at the R2RL Workshop at CoRL 2026. 🔗 r2rl-workshop.github.io/
No senior-speakers, no passive audience.
Key Q: What are the real bottlenecks in Sample-efficient, Real-World RL?
🧑🔬 No senior speakers. Students & scientists who actually deploy RL on robots.
🗳️ Crowdsourced problems. Audience proposes challenges, we all vote on what to focus on.
🧩Breakout sessions. Audience helps to propose and discuss solutions for top-voted problems.
🎤 Panel. The breakout discussions determine what we debate.
📝 Whitepaper: a community-constructed whitepaper summarizing the most urgent challenges for Real-World RL in the next year.
Submit your papers!
Attend to help us figure out what we should actually be solving next!
Submission deadline: September 30
Philip Schroeder retweeted
1. Robot data isn't the only way models can learn to control robots. Coding, math, computer-use all contain information about the physical world.
I bet that generalist LLMs will work better than robot-specific models for this reason
2. The output format (eg. joint angles, code-as-policy, tool calls as in Gemini ER) is up to debate. But why not just let an LLM choose?
Philip Schroeder retweeted
One thing I really like about the @pollenrobotics Microduck design are the legs!
I’ve labeled my guesses about where the joint axes are on two images, with red/blue/green to group axis alignments.
For each leg, starting from top:
- The hip is 3 DoF: yaw (red), then roll (I.e., abduction, green), then pitch (flexion, blue).
- The knee and ankle are each 1DoF (flexion, blue)).
It’s fun to compare the 🦆design to many humanoid designs that instead have flexion higher up the kinematic chain!
Philip Schroeder retweeted
"The generalization you want in your VLA has to be in the expert demos!" folks say.
"Didn't we train VLM backbones on trillions of tokens already for it?", we ask.
We set out to reclaim some of this (free) OOD generalization in VLAs. Just need to think beyond BC! 🥲
anchoralignvla.github.io
Two edits on top of BC. Same demos. No extra data.
Philip Schroeder retweeted
This cube visualization is my favorite way of remembering how batch norm, layer norm, instance norm, and group norm relate to each other!
This figure is from the original “Group Normalization” paper.
underrated observation in RL: learning what states are worth (a hard problem) can make learning which actions to take (a very, very hard problem) dramatically easier
In Reinforcement Learning (RL), the advantage function is a useful tool for improving policies!
I really love this proof that advantage is equal to expected TD error!
It's super useful in RL, because we can estimate the advantage (which depends on action) from just a state-value function (which is independent of action) and one-step transitions.
It's much easier to learn V(s) than Q(s,a), since the former is just a statistic of the latter (given the policy).
Really important for PPO when using Generalized Advantage Estimates (GAE)!
