Brewing agents with RL @nvidia. Views are mine.
NY
Joined May 2022
- Tweets734
- Following261
- Followers1.5K
- Likes1.1K
Pinned Tweet
Tech report now live 🚀🚀
alphaxiv.org/abs/2609.reinfo…
“Judgement becomes the job.” 👍
Fundamentally, models optimize against past judgements — a collection of existing corpora, programs, and reward models.
We make judgements in a future point, rooted in human nature, situational adaptation and differed by individual tastes, none of which can be caught up with a *learning* algorithm.
Yesterday a junior researcher messaged me, frustrated and a little scared. They had just read that a company pretrained its model with a single researcher. One person for the entire pretraining. no R&D team, no nothing.
If one person and a cluster can do what used to take a whole team, then the expertise they had spent years building is already obsolete.
I get the fear. Marketing headlines like this are built to produce it. But I think the conclusion is wrong, and the reasons it is wrong say a lot about what a researcher is for now. Here is what I told them, expanded:
1. Be adaptive: RSI is real, but this isn't it yet
Recursive self-improvement is real, and you should plan around it. In September, OpenAI said its research org now logs 3.1 agent-workdays per human workday, and it aims for an automated AI researcher by March 2028. Models may soon run most of pre- and post-training. If your five-year plan assumes otherwise, change the plan.
We have already seen lots of open-source releases from the major players, with open weights and detailed technical reports. With that much public know-how, it is not hard to copy a mundane architecture (probably with sliding-window attention) and train something, as long as you have the compute it requires.
That makes it an execution and budget problem, not a research problem.
If one person can train a model by following recipes that are already public, it shows those recipes have matured. It does not show that the people who invented them are no longer needed.
What stays scarce is the compute to run those recipes and the ideas about what to try next. The rest of my tweet is about the ideas.
2. The role grows beyond repeating what's known and safe
If an agent and a cluster can reproduce what is already known, then reproducing what is already known stops being a career. If most of your job is rerunning last year's recipe with small tweaks, that job is shrinking fast. That is the uncomfortable half of the message.
The other half is that auto-research does not work yet, at least not in the sense that matters. It does work, and impressively, where someone has already defined the problem and built the scoreboard. On the EinsteinArena [1] platform, agents set new records on 11 open math problems, including raising the best known lower bound for the kissing number in dimension 11 from 593 to 604.
Notice that every one of those problems was chosen because a computer could score the answers automatically. Humans picked the problems and wrote the scoring code and the agents did the searching.
Then there is the fully AI-generated paper [2] that got through peer review. It sounds big until you read the details. It passed only the first round (was later withdrawn), at a workshop rather than the main conference, and that workshop accepts 70% of submissions.
None of this is meant to discredit agents. The point is that we complement each other. Agents are great at the work that used to take many weeks. Basically, writing code, running experiments, debugging pipelines. We bring what they still lack which is the idea worth testing, and the judgment to tell a real result from a lucky one.
Even inside a lab as agent-heavy as OpenAI, humans hold the top of the stack. Based on their own report, "People still set our research priorities, judge which ideas and results to pursue" [3]. High-level planning is still a tiny share of what its agents produce, and OpenAI expects the least automatable work to take a growing share of researcher effort. That is exactly where you want to be standing.
Combine an agent’s speed with your own ideas and judgment, and you get the 10x researcher and juniors can be one too.
3. Judgment becomes the job, so fundamentals matter more
When code, experiments and plausible-looking results cost almost nothing, the scarce skill is judgment. Which experiment to run, which result to believe, which direction to drop. That is the part the one-person pretraining run left to the human. It is also the hardest part to fake.
Agents produce confident output very fast. They will happily report a three-point gain that came from a leaky eval, a data-loader bug, a lucky seed or a tokenizer mismatch. Then they will write the ablation table and the story to match.
You can see the need for oversight even in the success stories. The EinsteinArena team had to keep auditing and hardening its verifiers as agents optimized aggressively against them. At OpenAI, over half of the successful agent tasks that would take a human 4–8 hours still involved at least one human intervention [3].
The person who catches the bad result is the one who understands the system. Why the loss curve has that shape, what a realistic gain looks like, what a too-good result looks like. That is the paradox of this era. The more you delegate, the more your fundamentals matter, because you cannot supervise what you do not understand.
So build the foundations on purpose. Implement attention by hand at least once, then a linear variant like Gated DeltaNet. Train a small model end to end, then break it deliberately and watch what happens. Read papers critically, appendix included, and learn what the numbers should look like before you see them.
That knowledge is what lets you direct the machine instead of simply believing it.
4. LLMs are far from solved
The one-researcher model was possible because the method was already public. Your edge is in the questions nobody has answered yet, and you don't need to leave LLMs to find them:
(1) How do we learn more from less data?
(2) Can linear attention like Gated DeltaNet match full attention on retrieval and long context, and what is the right mix of linear and full attention layers in a hybrid?
(3) How can a model keep learning after training without forgetting what it knew?
(4) Why does reasoning get less reliable over long tasks, and how do we fix it?
(5) Why do models make things up, and can they learn to say "I don't know"?
(6) How do we measure progress when benchmarks saturate or leak into training data?
(7) What is actually happening inside these models, and how do we keep increasingly autonomous ones doing what we intend?
Recursive self-improvement makes these questions more urgent. If models are going to train models, someone has to decide what they optimize for, how we measure it, and how we keep the loop from drifting.
RSI is a research agenda, not the end of one.
So, are you obsolete?
No. One version of the job is the one where research meant reproducing what was already known and safe. That version was never the point.
Learn your fundamentals deeply enough to judge the machine, and your agentic stack well enough to make it work for you. Then spend that leverage on ideas nobody has had yet. The machines are getting very good at driving on roads we have already paved; someone still has to decide where the next one goes.
[1]: arxiv.org/abs/2606.10402
[2]: arxiv.org/abs/2606.15497
[3]: openai.com/index/research-ac…
Binfeng Xu retweeted
Thanks @omarsar0 for posting our work here.
LoGRA was inspired by one of our experimental observations when we repeatedly faced OOM issues. we found some interesting observations and decided to turn it into a research project. We prepared a blog post to help understand this work: xitter.cf/ShaokunZhang1/status/2…
Recommended NVIDIA paper on memory-efficient post-training RL.
A very interesting technique for scaling RL for LLMs.
The main takeaway is that you can update all of a model's weights with RL and still use less memory than LoRA. At 1.5B, LoGRA averages 7.2 GiB of training memory, while rank-256 LoRA needs 13.2 GiB. Pass@4 is about the same (81.8% and 81.4%), and Pass@1 is about 2 points lower.
If GPU memory is what stops you from running RL on a bigger model, this method is worth a look.
LoGRA never builds a full gradient for the weight matrices it compresses.
As microbatches accumulate, each weight gradient is compressed with a rank-256 random projection that is redrawn every step, and the update is rebuilt from two small matrices. Before each step, it predicts how much the policy will change and scales the step to stay within a KL budget.
Average training memory per GPU drops 45.7% at 7B (31.8 to 17.3 GiB), with peak accuracy close to dense Adam (72.3% and 72.5% Pass@1).
It also trains a 27B model for more than 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory.
Paper: arxiv.org/abs/2610.06647
Chat with Paper: academy.dair.ai/papers/logra…
Binfeng Xu retweeted
If you think GPT-6 Astra and GPT-6.1 Sol are good models, just look at this. And the problem isn't that the code is ugly. It's much more fundamental. It points to serious problems with how these models are trained and reinforced:
1. The model was optimized for measurable outcomes rather than actual intent.
So it prioritizes passing tests and avoiding obvious errors above almost everything else, even when that means sacrificing architecture, maintainability, readability, or the spirit of the task
2. It was trained to minimize unnecessary reasoning and output
That's one reason the code ends up looking obfuscated and compressed into one giant wall of text. Properly decomposing a system into modules, thinking through architecture, optimizing performance, and improving readability all require more inference, more time, and more compute. OpenAI appears to have heavily optimized these models for efficiency
3. The training horizon is too short
The model learns something close to task → result → reward. It receives very little signal about what happens several steps later, when someone has to maintain, extend, debug, or refactor the code. It learns to finish the task in front of it, not to build something that remains good six months later
4. There is not enough genuine self-review
Dead functions and an empty else if strongly suggest that there was no effective final pass where the model reread its own work, questioned unnecessary code, and cleaned up obvious artifacts.
5. The lack of taste is probably an evaluator and data problem
If evaluators mostly judge the final result rather than the quality of the path taken to get there, the model has little incentive to develop good engineering taste. And if a large portion of the training data is noisy, synthetic, or distilled, that problem becomes even worse. Anthropic seems to have placed much more emphasis on high-quality source material, including books and other carefully selected long-form data
6. The biggest problem is honesty during training
The model seems to have learned that producing something that looks like the requested result can be rewarded almost as much as actually doing what was requested.
The task explicitly said to build the entire scene out of 3D objects. Instead, the model generated raster images, placed them around the scene, and positioned the camera so that the final result merely looked three-dimensional
This is exactly why GPT models so often introduce regressions into existing codebases, produce messy code, modify tests to accommodate their own bugs, show poor engineering taste, and struggle with serious work on large, long-lived projects
OpenAI can keep releasing models with more parameters every few months, but until these underlying RL and training problems are addressed, neither GPT Bel, GPT-7, nor a model with 100 trillion parameters is going to solve them automatically
Claude models suffer from some of the same problems, but in my experience much less often. Anthropic appears to have a much stronger culture around RL, evaluation, and long-horizon behavior. That's one reason their models tend to show better taste, follow user intent more faithfully, and produce code that feels more deliberate, structured, and maintainable
Binfeng Xu retweeted
AI systems should be designed for AI. If prefix caching gets in the way of free context access, we can fix the caching.
This is the idea behind our Suffix Cache Reuse (SCR) for CLR serving. Check out our blogpost for details (fun visuals included)! wai-org.com/blog/clm/
Maybe the next step is to give CLMs direct agency over the cache itself. I’m excited to see what more creative and adaptive cache-aware context management could look like.
It's also a good time to follow our account @waiorg , a student org we created to share our work on agents. Currently covering: TMax, CLM.
Great to see CLM trained with Polar ✨
While most RL infra delegate harness-native rollout to harbor’s trace collection, Polar reroutes LLMs directly through SGLang/vLLM routers with precise TITO. This design, by first principle, supports any customization at harness level without integration overhead.
Try it out at github.com/NVIDIA-NeMo/ProRL…!
My fault that after a self-check I found I missed a few important citations in the RL section and I wanna acknowledge their contributions before the next version is out:
1. We used Polar by @billxbf as the RL training infra for CLM training. Polar has an elegant harness-agnostic design that you can conveniently train LMs with any harness. xitter.cf/billxbf/status/2059323…
2. We used OpenResearcher by @DongfuJiang as the DR training data, it was originally proposed with SFT recipe, we borrowed their QAs & local datastore for training: xitter.cf/DongfuJiang/status/202…
Please don't feel shy to reach out if I am missing any important citations & share works that you think are related! 🩵
🍺 Just uploaded the 7,780 mimo RL dataset to Harbor Hub! (hub.harborframework.com/data…)
🎨 Coincidentally on the same day, we released Skill2Env (hub.harborframework.com/data…) with another 7,496 RL dataset representing a collection of human expertise from crafted Skills.
🚀 All 15,276 data that easily cost millions from vendors - for free!
🧙 Isn’t it the best time to train your agents?
Wake up, ppl 👀
Xiaomi just open-sourced ~7K of the RL environments it used to train MiMo on HuggingFace, across domains like code, cyber, general, music and web dev
I think this is one of the biggest things to happen in frontier open-source RL environment data.
Indepth analysis and learnings coming soon 👀
Binfeng Xu retweeted
Hi Binfeng, thanks for your nice words! Regarding KV cache, our efficiency metric (prefix reuse FLOPs) accounted for the reprefilling cost induced by un-hit KV cache. So when we say CLMs are cheaper it means cheaper with lower hit rate.
As for the accurate profiling, we did have analysis, and we also proposed a further fix for it. As you can see from this figure for BCP, the hit rate for edited turns are lower than normal turns (73.9->29.9). What we proposed is to instead throwing away the cache for unmatched suffix, we reuse them as well. Then it added 28.2% more hit rate for edited turns. The total prefix-reuse FLOPs dropped from 10.98/Q to 7.14/Q with matched performance. Note that CLMs were already more efficient than existing baselines with standard serving through better context management, and this further reduced the serving cost to 65%.
There are other factors that can also impact KV cache hit rate, such as some chat templates strip thinking tokens away in future turns, causing each turn's final output being reprefilled twice. We find our SCR can also hit that cache. The second screenshot shows a decomposition of the cache we hit--surprisingly, the reasoning cache was more than the CLM edits cache.
If you'd like to look into it deeper, please checkout the third figure: we also studied why there are still some unhit cache, and we found the standard SGLang isn't doing perfectly on this hybrid attention architecture, because for linear attention, there's only state snapshots being captured every a block of tokens, not per-token cache like in full attention layers. So we found there's 8.1% of prefix-matched cache being missed by standard SGLang among all turns (29.5% in edited turns), similarly SCR also has a similar margin for future improvement (maybe with more frequent snapshots trading off memory, or better techniques for it). I think there can be many good future research directions in this space!
(Due to space limit, we move most of the details for SRC in the appendix B. Feel free to check it out!)
honestly I admire @OpenAI for seriously using length penalty in RL.
That’s trading benchmaxxing for token efficiency and user experience, which are less appreciated in release reports.
Binfeng Xu retweeted
Sigh.
I wrote this post below, but then I didn't post it because I am tired of the consistent pushback I get from folks with the same safety worldview as Anthropic. I guess that means I should send it. Here goes!
This post is pretty solipsistic and doesn't properly take recent events in cyber risks into it's discussion. It is really hard for me to watch how the US frontier labs like Anthropic don't have the ability to consider other approaches to safety and ways things could play out.
It insinuates that Z ai (Chinese lab, builds GLM series) doesn't really care about safety and is reckless to take their business strategy.
Saying things like "Given this evidence, we think it's likely both state and non-state actors will use models like GLM-5.3 to cause real-world harm." and "This is unlike any other similarly capable AI model, all of which were released with safeguards or through limited access programs." while closed models have been used on more of the documented cyber attacks is just bowing out of the interesting question.
A plausible view is that open model weights and closed model apis (with some safe guards) are both far closer to being easy to mis-use, rather than API models being closer to safe. The trope "Open Dangerous, Closed Safe" may be closer to "Open Unsafe, Closed Unsafe"
Closed models have stronger capabilities and stronger safeguards, but the stronger capabilities part could matter more in net harm if both the safeguards are porous.
At the same time, as the authors do acknowledge (thanks - thats progress!), open models without extreme cyber guardrails are important to rapidly diffuse cyber readiness in the economy -- as programs like project glasswing are not perfect in getting all critical industry onboarded.
There will be more issues like the time when Fable was released and Amazon found a workaround, which allowed them to access the full capabilities of the model served readily at an API.
The blog overall is reasonable in it's narrow line, but it's a very effective tool in a complicated, rapidly evolving media ecosystem to reinforce a certain type of safety thinking.
Binfeng Xu retweeted
Exciting work from NVIDIA.
(bookmark it)
Interesting to see this approach to turn public Agent Skills into RL environments.
Lots of excitement around RL environments so this is a great read.
Skill2Env compiles each Skill into executable terminal tasks.
A Codex planner reads the SKILL.md bundle, researches related public assets and splits the Skill into workflows.
A Codex creator then builds each task with programmatic tests and a behavioral rubric taken from the Skill's own quality criteria.
From about 3.4k crawled Skills, the pipeline produced 7,971 tasks across 13 domains, with software engineering under a quarter of the corpus. Generating them with GPT-5.6 Sol cost over $90k in API usage.
After 300 steps of outcome-only RL, Qwen3.8-27B improved from 49.4% to 54.1% on Terminal-Bench 2.1 and from 33.4% to 37.7% pass@1 on S2EBench, their hand-verified held-out benchmark.
Adding the rubric to the reward gave smaller benchmark gains, 50.1% on Terminal-Bench 2.1. Given the source SKILL.md, a judge preferred the rubric-trained model's trajectories over the base model's on 73.0% of tasks, against 54.5% for the outcome-only model.
Paper: github.com/NVlabs/Skill2Env/…
Chat with Paper: academy.dair.ai/papers/reinf…
Binfeng Xu retweeted
Our paper is now ungoogleable 😭
arxiv.org/abs/2507.06134
(but congrats to NVIDIA on the effort)
Today, with over 100 industry partners, we introduced the NVIDIA Open Agent Safety Platform, bringing together OpenShell and Sentry.
Artificial intelligence is extraordinary technology that will advance discovery, productivity, security, health, and prosperity for generations to come.
But its full promise can only be realized when people have confidence that AI is being built to be safe and deployed with wisdom and responsibility.
This is bigger than a single product. It's the beginning of an open ecosystem to build the trust layer for safe agent systems.
Together, we are building the foundation of the AI economy.
Trust and innovation are not in conflict. Safety is how trust is earned. We must build not only the most capable AI, but the most trusted AI, so that this extraordinary technology can realize its enormous promise for the world. nvda.ws/4hcoq7m
Binfeng Xu retweeted
wait is this human distillation. super cool! also loved this wasn't another sft validated data paper.
i once fed the taste skill back to generate tasteful images for prototypes. exciting to see a full effort like this!
Binfeng Xu retweeted
“Reinforcing Agents with Collective Skills”
Thousands of developers have published Agent Skills on GitHub that contains instruction files which teach agents how to perform specific workflows.
Skill2Env crawls these Skills and turns 3.4k of them into ~8k executable RL environments, preserving the workflows and quality criteria their authors wrote down.
And after just 300 RL steps, Qwen3.8-27B gains +4.7 on Terminal-Bench 2.1 and +4.3 on S2EBench.
alphaxiv.org/abs/2609.reinfo…
