MS @CarnegieMellon Undergrad @iitbombay
Pittsburgh
Joined September 2021
- Tweets172
- Following1.8K
- Followers420
- Likes11.3K
Pinned Tweet
Is RL optimizing the right objective? 🤔
Should we maximize mean reward? Best-of-k? Which k?
Standard RL pulls on the mean and often the distribution collapses to a spike. The tail dies 🥲
We introduce Tail-Likelihood Reinforcement Learning (TailRL).
It maximizes the mean reward while simultaneously maximizing coverage over high reward outputs.
🧵 1/n
Shrinivas Ramasubramanian retweeted
Check out ReSYNC!
Robots often fail. ReSYNC not only helps them recover but also discovers concepts to explain and avoid failures during unseen long-horizon tasks.
Failures are inevitable for robots that are deployed in the *big* world. We introduce ReSYNC, a system that leverages large-scale RL sim2real to recover from local failures and concept discovery to explain/avoid new failures. Check it out here: jaraxxus-me.github.io/ReSYNC…
All of us have experienced existing language models making stuff up and give wrong answers on hard problems.
An important feature of language models is that they know they don't know the answers and verbalize it.
How about we train the language model to do so...
Check out Schema, new work from @guanningzeng @Zanette_ai and @HavenFeng! It lets frontier LLMs direct their own learning by building and refining an executable theory, testing it against experience, and using it to explore and plan.
On July 15, we released Schema, the first agent harness to reach ~99% RHAE on the ARC-AGI-3 public set.
Today, we’re sharing the full paper and code, with new results and a closer look at why Schema works.
xitter.cf/HavenFeng/status/20777…
Shrinivas Ramasubramanian retweeted
On July 15, we released Schema, the first agent harness to reach ~99% RHAE on the ARC-AGI-3 public set.
Today, we’re sharing the full paper and code, with new results and a closer look at why Schema works.
xitter.cf/HavenFeng/status/20777…
Shrinivas Ramasubramanian retweeted
Delighted to receive an ✨Outstanding Paper✨ at #COLM2026 for our paper “Message Passing Enables Efficient Reasoning.”
This work was done w Xuecheng Liu, @g_k_swamy and the 🐐 @Zanette_ai
Congratulations to the authors of the 3 outstanding papers!
arxiv.org/abs/2607.01077
arxiv.org/abs/2604.08510
arxiv.org/abs/2604.13977
I will be giving a talk on our recent work Tail-Likelihood Reinforcement Learning on clubhouse at 7PM EST (4PM PDT). Please feel free to join us!
Join @stablegradients of @CarnegieMellon
om a deep dive with @ceciletamura of the @qpclub1
at 🎧 @Clubhouse
clubhouse.com/invite/QK7f20A…
Shrinivas Ramasubramanian retweeted
Ask any top image model for "a dog with a purple left ear and a striped tail" and watch it struggle. 🐕
Part-level control is still an open problem for NB2, Flux & co. Our @NeurIPSConf paper tackles it head-on.
Amazing collaboration with @vikataravi @VarghPK @va1bhavagrawal 🙌
Excited to share PartConcepts, a unified mechanism for fine-grained part localization and generation, accepted to #NeurIPS 2026 @NeurIPSConf 🥳🥳🥳
🌐visionpartconcepts.github.io
Shrinivas Ramasubramanian retweeted
Heading to Sydney for #Interspeech2026! 🐨
Excited to present our SOTA universal phone recognition model in the oral session:
Tuesday 29 September - 16:30-18:30 - Area 9 - Oral 1
Code and models: github.com/changelinglab/Pho…
If you are at ECCV, you dont want to miss this! We finally have visual representation downstream of text guidance. Talk at 9AM Malmo arena.
Pretrained ViTs like DINOv2 or CLIP are great, but they produce fixed, generic representations that encode the most salient visual concepts (e.g., "cat").
In human vision, prior priming with language changes how people parse an image. We believe visual encoders should do the same
🚨 Introducing Steerable Visual Representations, a new family of visual features you can steer with text towards specific visual concepts.
Shrinivas Ramasubramanian retweeted
Check out our new work on Tail-Likelihood Reinforcement Learning (TailRL), extending maximum-likelihood RL from binary to continuous rewards.
arxiv.org/abs/2609.02987
Rather than optimizing only mean reward, TailRL maximizes the expected log of upper-tail probabilities, naturally placing more weight on rare, high-reward rollouts. Its gradient can also be interpreted as a mixture of Best-of-(k) gradients.
TailRL requires only a simple modification to the advantage function, making it easy to integrate into existing RL pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL effectively exploits rare high-reward samples and scales better with increased inference-time sampling.
Check out a detailed thread by @stablegradients.
Is RL optimizing the right objective? 🤔
Should we maximize mean reward? Best-of-k? Which k?
Standard RL pulls on the mean and often the distribution collapses to a spike. The tail dies 🥲
We introduce Tail-Likelihood Reinforcement Learning (TailRL).
It maximizes the mean reward while simultaneously maximizing coverage over high reward outputs.
🧵 1/n
Please check out our new work Tail-Likelihood Reinforcement Learning on @askalphaxiv
“Tail-Likelihood Reinforcement Learning”
Standard RL optimizes average reward, which can suppress rare but exceptional outputs and hurt Best-of-k scaling.
This paper instead optimizes the probability of exceeding every reward threshold, giving more weight to rare high-reward rollouts.
Its gradient becomes a harmonic mixture of Best-of-k objectives across all k, helping preserve exploration and inference-time scaling, with up to 256x fewer samples on GUI grounding and 7.7x Best-of-1024 speedup on code optimization.
alphaxiv.org/abs/2609.02987
Shrinivas Ramasubramanian retweeted
The most common question we got after MaxRL was: “What if the rewards are continuous?”
I think we now have a pretty good answer. Check out Tail Likelihood Reinforcement Learning (TailRL), which extends MaxRL to continuous rewards.
Much like MaxRL, we’ve found that TailRL can “get you out of trouble” on reinforcement learning problems that are particularly difficult to optimize, while maintaining much better coverage of rare, high-reward solutions, which is useful for preserving diversity and improving test-time scaling.
Curious? Check out the thread below by @stablegradients.
Is RL optimizing the right objective? 🤔
Should we maximize mean reward? Best-of-k? Which k?
Standard RL pulls on the mean and often the distribution collapses to a spike. The tail dies 🥲
We introduce Tail-Likelihood Reinforcement Learning (TailRL).
It maximizes the mean reward while simultaneously maximizing coverage over high reward outputs.
🧵 1/n
The best part of research is the friends you make along the way. I am extremely grateful to my friends and co-authors
@amuseddaman , @FahimTajwar10 and @guanningzeng
for their contribution and constant support. This project would not have come to fruition without the mentorship of @yus167 , Drew Bagnell, @rsalakhu , @Chenfeng_X , @ZhongzhuZhou , @QingyangWu1 , @HavenFeng and Aarti Singh.
A huuugee shoutout to my advisors. The best advisors one could imagine!!
@Zanette_ai and Jeff Schneider.
🧵 13/n
Please also see this great work from @_doubleAI_ @NadavSchweiger which also arrived at the same objective, I guess great minds think alike !!
It is such a joy to have your work validated independently.