Pinned Tweet
🪂Understanding R1-Zero-Like Training: A Critical Perspective
* DeepSeek-V3-Base already exhibits "Aha moment" before RL-tuning??
* The ever-increasing output length in RL-tuning might be due to a BIAS in GRPO??
* Getting GRPO Done Right, we achieve a 7B AIME sota!
🧵
📜Full details: github.com/sail-sg/understan…
🛠️Code: github.com/sail-sg/understan…
Nothing is more exciting than the progress of the last 2 weeks, from Argon to Carbon.
What a time to be working on Gemini!
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model.
Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching.
We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards.
As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We'll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
Zichen Liu retweeted
Gemini 4 Carbon is the best frontier model right now! 😎
Sota on AA index and a bunch of other stuff! 😎
Exciting times!
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model.
Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys Intelligence Index at 59. It also leads across key cybersecurity evaluations scoring 88.2% on SEC-bench Pro for finding vulnerabilities, 54.8% on ExploitGym for exploitation, and 71.0% on CWE-bench for patching.
We’re seeing significant improvements in long-horizon tasks even compared to Argon, and we’ve strengthened frontier safeguards.
As we did with Argon, we’re going to roll it out gradually + responsibly. Some trusted cyber defenders will get access starting this week. We'll be expanding it to more partners through our Fairwind Program, ahead of wider availability soon, so hang tight!
It’s an awesome co-worker, and hope you enjoy Argon as much as we do!
Zichen Liu retweeted
Gemini 4 is here! 🚀 And it outperforms astra/fable on many things 😇🤓
Zichen Liu retweeted
HyCreator: an agent harness for long video generation
🚀 End2end Auto Mode
⏲️ Generate 10-minute scale films with zero human intervention
👨🦰 Switch to real-time interactive editing whenever you want
Apply for early access: hycreator.tencent.com/
Zichen Liu retweeted
🚀 Will Agentic ESOpt Be the Future of Long-Horizon Agentic Fine-Tuning?
Excited to share our new work on Arxiv.
Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Memory Requirements
As LLM agents become longer-horizon and more branching, conventional Agentic RL faces three growing bottlenecks: high training/memory cost and increasingly challenging long-horizon credit assignment.
Our Agentic ESOpt approaches agent fine-tuning from the parameter space instead, targeting three advantages:
⚡ Model Scalability — only minimal inference-level GPU memory requirement
🔧 Flexibility — prompt/skill–parameter co-evolution
⏳ Long-Horizon Scalability — trajectory-level parameter attribution without horizon-wise decomposition
There are impressive empirical results, and we believe ES may be a better-matched parameter fine-tuning paradigm as agents become larger and longer-horizon.
📄 Paper: arxiv.org/abs/2608.17310
🌐 Project: zz1358m.github.io/Project-Ag…
💻 GitHub: github.com/zz1358m/Agentic-E…
🤗 Models & Checkpoints: huggingface.co/collections/z…
Code, training logs, and checkpoints have been fully open-sourced.
Main takeaways ↓
Zichen Liu retweeted
Continuous self-improvement needs an ever-expanding supply of training environments (goals).
SPADE: one model self-plays the Environment Designer and the Reasoning Agent, writing executable, agentic environments that get harder as it improves. Environment scaling on its own. ♠️
Zichen Liu retweeted
PI is cooking again, multi-agent RL support just landed in verifiers + prime-rl.
Glad to see SPIRAL's Role-conditioned Advantage Estimation (RAE) featured and implemented for self-play!
Today, we’re extending our RL stack beyond individual agents to multi-agent systems.
You can now express arbitrary agent interactions and train them.
primeintellect.ai/blog/multi…
Welcome submissions if you work on meta-agents!
🚀Call for Papers — @NeurIPSConf 2026 Workshop Workshop on Responsible Use of Meta-Agents
📅 December 11/12 · 📍 Sydney, Australia
Join us to shape the future of Meta-Agents and Harness. Topics include but are not limited to:
1. Automated Design of Agent Harnesses
2. Optimization of Agentic Systems
3. Recursive Self-Improvement
4. Evaluation and Benchmarks for Meta-Agents
5. Misalignment and Safety for Meta-Agents
6. Governance & Human Oversight of Agents
🌟 Featuring an all-star speaker lineup:
Graham Neubig @gneubig (CMU)
Chelsea Finn @chelseabfinn (Stanford)
Jenny Zhang @jennyzhangzt (Recursive)
Robert Lange @RobertTLange (Sakana AI)
Bo Li @uiuc_aisecure (UIUC)
Hancheng Cao @CaoHancheng (Emory)
Yuandong Tian @yuandongtian (Recursive)
Chen Sun @ChenSun92 (Google DeepMind)
Proudly sponsor by @metaai and @e2b
Updates: Follow us here & spread the word! #NeurIPS2026 #MetaAgents #LLMs #Harness #RSI #SelfImprovement
Zichen Liu retweeted
Announcing Discovery Loop!
I am very excited to announce that, along with my longtime friends and collaborators @Sanjay_Ghemawat, @OriolVinyalsML and @quocleix, we are founding Discovery Loop (@DiscoLoopAI), a Public Benefit Corporation whose mission is to automate machine learning, science, and engineering to accelerate discoveries and progress. The four of us have worked together for 14 to 30 years, and have helped build some of the world’s most used products, infrastructure and AI models, and we’re excited to turn our attention to this ambitious endeavor.
♾
Learn more at: discoveryloop.com
Zichen Liu retweeted
A few months ago, @qx_dong turned me from a pure IC into a manager. 🤗
Was a really fun experience so far watching people join one by one, building the team culture together and going up on the frontline. 🫠
Unprecedented moment in the history of Singapore to have so many talented researchers in one place. I'm glad to be building this legendary team. 🫡
Overdue update: I’ve joined @GoogleDeepMind as a research scientist to work on 💎Gemini post-training.
Feeling incredibly fortunate to be cooking models with such a brilliant team, under the leadership of Yi Tay (@YiTayML) and Quoc Le (@quocleix).
In the past a few month, we've found so much enjoyment in working with our own models and keeping pushing their boundaries. Ps, there's a special joy in creating recipes and internally naming them after our favorite drinks🧋
Looking forward to keep enjoying research, pushing the frontiers of Gemini and seeing the magic unfold on the path to AGI!
Zichen Liu retweeted
Our latest deliveries:
👨💻 UniRL, an RL infra for unified multimodal models
➕ Two new RL algorithms:
• FlowDPPO for diffusion and flow-matching models
• DRPO for LLMs and VLMs
Check out our repo: github.com/Tencent-Hunyuan/U…
🚀Introducing UniRL, an RL infra for unified multimodal models. Together with two new RL algorithms: DRPO and Flow-DPPO.
One RL loop across diffusion/flow matching models, LLMs/VLMs, and unified multimodal models👇
Code: github.com/Tencent-Hunyuan/U…
(yes — U(you)-ni-(need) RL 😉)
Zichen Liu retweeted
We propose DRPO: a soft version of DPPO🔥
Since PPO, clipping/mask-based trust regions have long outperformed smooth divergence regularization like KL, even though the latter one feels more principled. 👺
We found two missing pieces:👇
1️⃣ Weight the regularizer by |advantage|
- Otherwise, the trust region geometry changes dynamically and optimization becomes unstable.
2️⃣ Use the right divergence
- What matters is not just “regularization”, but the trust-region geometry induced by the gradient. DPPO-style geometry works much better than PPO-style geometry in LLM.
These insights lead to DRPO, which delivers the most robust and best overall performance across algorithms, even outperforming original mask-based DPPO. 🚀
This project is an amazing collaboration with @ExplainMiracles, @NickZhou523786, Wee Sun Lee, Liefeng Bo, @TianyuPang1 . Do follow them if you are interested in this work!
📄 Paper: arxiv.org/pdf/2606.09821
💻 Code: github.com/Tencent-Hunyuan/U…
Zichen Liu retweeted
🧵 For 2 RL checkpoints trained differently, you can just weight extrapolate them and it works!
Bonus: these extrapolated checkpoints are complementary policies
-> Get exploration and diversity for free
-> Better inference scaling when ensembling
Paper: arxiv.org/abs/2605.28751
arxiv.org/abs/2605.28751
Now many studies try to do extrapolation through model merging. arxiv.org/abs/2605.26484
Zichen Liu retweeted
Today at Google I/O, we introduced Gemini 3.5 Flash! It has become an integral part of our daily research cycle and works with all the tools we have at Google.
We used a team of agents in Antigravity 2.0 to recreate the original AlphaZero research paper and build a playable version. They coded the reinforcement learning pipeline in JAX/Flax, trained a ResNet model from scratch via self-play on multi-TPU pods, and shipped a full-stack web app so you can play against it, from just 2 prompts. .
Here’s what else makes 3.5 Flash special 🧵
rl intuition (up-scaled by the correctness of infra) is all you need when cooking with a strong base model such as gemini✨
Zichen Liu retweeted
🚀 Excited to share RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback! Instead of training LLM agents just to solve isolated tasks, we designed them to continuously evolve.
By using retrospective dual intrinsic feedback (Numerical for exploration + Language for explicit memory) & SimUtil-UCB retrieval, we achieve massive SOTA gains over GRPO baselines:
🔥 +18.3% on ALFWorld
🛒 +15.4% on WebShop
📦 +27.1% on Sokoban
💣 +8.9% on MineSweeper
Strong test-time adaptation & OOD generalization included! 🧠👇 📄 Paper: arxiv.org/abs/2603.08561💻 Code: github.com/zhangxy-2019/Retr…
Zichen Liu retweeted
Text agents have their Gym. Vision agents? Not until now.
Introducing Gym-V — a unified gym-style platform for agentic vision research, with 179 procedurally generated environments across 10 domains.
One API to rule them all:
📦 Offline dataset
🤖 Agentic RL training
🔧 Tool-use training
👥 Multi-agent training
📊 VLM & T2I model evaluation
All under the same reset/step interface.
Key findings:
1. Observation scaffolding matters MORE than RL algorithm choice
2. Broad curricula transfer well; narrow training causes negative transfer
3. Multi-turn interaction amplifies everything
📄 Paper: arxiv.org/abs/2603.15432
💻 Code: github.com/ModalMinds/gym-v
Open the thread for a deep dive! 🧵
