PhD @nyuniversity | Researcher @modal Prev @Columbia
Joined July 2024
- Tweets104
- Following521
- Followers324
- Likes304
Pinned Tweet
Scaling laws have transformed how we understand pretraining, and recent work has begun to characterize scaling in RL. But these stages are almost always studied in isolation. Can we study pretraining and RL jointly, and derive a unified scaling law for the full pipeline? 🧵
Leon retweeted
Will your ICLR paper get accepted? 👀
Introducing PaperScorer, our NeurIPS '26 project that predicts acceptance from anonymized submissions.
It beats GPT, Gemini, and Jev, with well-calibrated probabilities.
Try it on your paper today: paperscorer-ai.com
Blog: sachinkonan.github.io/PaperS…
Details in thread 🧵
Leon retweeted
✨ Introducing AutoRLBench — a benchmark built from real RL training failures.
AI training AI is no longer hypothetical. Agents already run post-training, read the result, and decide what to do next.
But a loop only turns if each round can be judged. When a run goes wrong, the agent has to find the root cause, and fix it — otherwise the loop runs on a broken signal.
AutoRLBench is the first benchmark that measures those abilities on real RL training failures. And we're recruiting more contributors 🧵
🌐 Project: thinking-space.github.io/Aut…
Leon retweeted
Tinker changed how people do post-training. We're big fans, and so are many of the teams we work with. But as those teams scale up, they keep raising the same two problems: per-token pricing gets expensive quickly, and you’re limited to the models and features the provider decides to support.
We think a good API deserves more than one implementation.
Today we're releasing Spindle, an open-source, Tinker-compatible training server that runs in your Modal workspace. Your existing Tinker scripts run unchanged, and because you own the server, you pay for GPU time instead of tokens.
Leon retweeted
All of my iclr review batch are quite slop. some interesting patterns of how AI slops:
1. agent tends to spend a shit ton of resource and paper real estate on ablations or experiments that are too minor to matter
2. likes to use a lot of numbers. real interesting point is buried. one abstract has like 15 numbers in it.
3. either overclaims something aren't true or be so paranoid that every claim comes with 3 disclaimers
I suspect a lot of them don't just use AI to write the manuscript but also use agent to come up with what to run or what direction to take. As a result, you get very rigorous tables but with no substance.
Leon retweeted
1/9 Current optimizers are static during training. What if the optimizer could think and react while the network trains?
We let an agent watch the training dynamics and rewrite the optimizer's source code as training goes: a reactive optimizer.
In every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself. In the near future, optimization decisions may belong to agents, not humans.
📝 xingjianbai.com/reactive-opt…
💻 Try it on your task: github.com/xingjian-bai/opal
🧵
Want to know who's the most cited person in ICLR 2027, and which submissions cite your paper?
I downloaded all 43,013 public ICLR 2027 submissions through the OpenReview API and asked my good friends Codex and Claude to build a site that ranks the most-cited papers and authors.
#1 paper is not Attention Is All You Need 👀
Link: modal-labs-leon-dev--iclr202…
Some interesting facts about titles and abstracts:
• 992 titles start with "Learning", 931 with "When", 865 with "Beyond"
• 600 different "…Bench" names
• 66% of titles contain a colon
• 929 titles end with a question mark
• 24 titles claim something "is all you need"
• Most used words in abstracts: "novel" (2,384), "paradigm" (2,335), "leverage" (1,416)
Leon retweeted
Come work with @Qi_Lei_ and me on AI self-improvement at NYU! We are looking for a postdoc, details: apply.interfolio.com/195745
We’re hiring a postdoc at NYU Courant Data Science! Join @Pavel_Izmailov and me to work on self-improving AI and AI for mathematics.
Two-year position with a flexible start in 2027-2028.
Apply: apply.interfolio.com/195745
Please share!
Leon retweeted
Today we're announcing that Modal has crossed $1B in annualized run rate, as well as raised a $750M Series D at a $15.75B valuation, led by @Accel.
Since our Series C, usage has tripled and we’ve grown more than 15x year-over-year.
We are deeply grateful to the developers and companies building on Modal. There's still so much more to build, and we can't think of a better time to be building software.
Leon retweeted
🚨 LLMs are trained on past data and deployed in the future, but we evaluate them like time doesn't exist. By treating LLM evaluation as a forecasting problem, we can better predict how models will actually perform in the future. 📈 1/n
Leon retweeted
Spent the past three weeks on this idea.
A sampler on a different precision or accelerator is just off-policy, like stale samples in fully async RL, and off-policy correction doesn't care why the sampler differs from the trainer. Meanwhile, inference keeps diversifying across precisions(MXFP8, MXFP4, NVFP4) and accelerators outside RL. So why not use all of that compute for RL rollout?
The sampler and the trainer are two systems built for different purposes. Hope this opens up new possibilities for RL algorithms and systems, and leads to cleaner system designs. Enjoy the blog, and stay tuned for the next one on our infra :)
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Leon retweeted
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Leon retweeted
Announcing Exploration-Distillation, a RL framework that unlocks aggressive exploration and large performance boosts over standard algorithms like DAPO. In principle, an LLM can discover new things during RL, but existing algorithms struggle. Here’s how we do it. 🧵1/n
Leon retweeted
Interactive software built on demand is becoming a dominant way we interact with AI.
But we often end up as its testers: we find a bug, tell Claude, wait for a fix, and repeat.
SWEeper-Bench 🧹 measures whether agents can sweep those bugs away before we ever see them. Surprisingly, Grok 4.6 does it best 🧵
Leon retweeted
How fast is AI's research taste improving?
We find that the experimental research taste of frontier models has doubled every ~3 months since December 2025. The best model, Opus 5.5, now exceeds our expert human baseline. Our human experts are experienced researchers, but most haven’t worked at a frontier lab.
Why measure research taste? In the AI Futures Model, it largely determines how quickly artificial superintelligence is reached once coding is fully automated.
