PhD student at @Columbia, collaborating with @MSFTResearch | Previsouly undergrad at @CUSEAS
Joined January 2023
- Tweets45
- Following139
- Followers87
- Likes18
Xiao Yu retweeted
Just read this new paper from Microsoft and i think this is gonna solve a lot of headache.
Anyone who's tried RL-tuning agents inside real inference setups (like Claude Code or OpenClaw) knows the pain.
Current open RL stacks just aren't built for stateful, multi-process harnesses, so you almost always end up with a nasty train-deploy mismatch.
Their fix is ridiculously simple in hindsight: just completely decouple inference from training.
They use a light proxy to log model calls during rollouts and throw each rollout into its own remote container using Kubernetes.
That data just streams back into standard RL frameworks like veRL.
No hacky workarounds.
And the results are wild for the amount of data used: just a few hundred to a couple thousand tasks and OpenForge-GUI is pulling 37.7 on OSWorld-Verified and 72 on WebVoyager.
It’s outperforming baselines of similar size and matching models way larger than it.
Also pretty interesting to see their analysis on behavior: RL clearly turns the dial on stuff like self-verification and tool coverage, but agents are still pretty awful at actual error recovery when things go off the rails.
Definitely give it a look if you're working on agent infrastructure.
Read the full paper here:
arxiv.org/pdf/2607.21557
Xiao Yu retweeted
New research with Microsoft's and colleagues on training agents inside the harnesses they actually run in.
(bookmark it)
Why it matters:
Agents today live inside elaborate harnesses like Claude Code, Codex, and OpenClaw.
Those harnesses are hard to train end to end because open RL stacks cannot express stateful, multi-process harness inference. Most training happens in stripped-down environments that look nothing like deployment.
OpenForgeRL closes that gap. A lightweight proxy serves the harness model calls while recording them as training data for a standard RL codebase, and a Kubernetes orchestrator runs each rollout in its own container. You train directly in the real harness, at scale.
Using only hundreds to a few thousand tasks, OpenForgeGUI reaches 72.3 on WebVoyager, 63.0 on Online-Mind2Web, and 37.7 on OSWorld-Verified, beating open baselines of similar size and matching models several times larger.
Harness choice turns out to be a training variable. Some harnesses are much harder to learn than others, and RL improves self-verification and multi-step completion while error recovery stays weak. That is a useful map for anyone doing agentic RL.
Paper: arxiv.org/abs/2607.21557
Learn to build effective AI agents in our academy: academy.dair.ai/
For AI agents to scale beyond narrow tasks, they need to learn how the world works from their own interactions — in a self-supervised way, without relying on expert data or rewards.
We introduce Reinforcement World Model Learning (RWML), a self-supervised method that trains LLM-based agents to align their simulated next states with actual environment dynamics, improving robustness and adaptability across tasks.
Empirically, we find RWML directly boosts solve rate on ALFWorld and τ² Bench without expert data — and scales naturally to large RL settings.
[4/n] Takeaway: agents can learn more from their own experience than we often assume.
By aligning imagined next states with what actually happens — and doing so in a representation space that highlights meaningful changes — RWML turns every interaction into a learning signal and builds richer world understanding.
Shout out to my collaborates at @Columbia @MSFTResearch @dartmouth for all the support!
paper link: arxiv.org/abs/2602.05842
Accepted by #ICLR2026 🎉See ya in Brazil!
Why can (V)LMs agents ace coding and math, yet struggle so badly in more complex environments like computer or phone use? 🤔
We find that one key factor lies in models' ability to understand and *simulate* the environment’s dynamics — and propose **Dyna-Mind** to address this!
🧵[1/n]
[4/n] Take-away: agents can learn not just from success, but from *every* environment interaction to enhance their reasoning/acting, if used correctly💡
Xiao Yu retweeted
Can we ever truly trust foundation models—and if so, how?
Our ICCV TrustFM workshop (t2fm-ws.github.io/T2FM-ICCV2…) is now accepting submissions (deadline: 8/1, attending: 10/19-10/23, Hawai'i)
Submit, attend, and learn from everyone around the world who is making FMs more trustworthy
Co-organized with Zeliang Zhang, @ChenliangXu, @huang_chao_cs, @MuCai7, @DrogoKhal4, @jd92wang, @hendrycks, @jure, etc.
@trustworthy_ml @StanfordAILab
Xiao Yu retweeted
🚨COLM 2025 Workshop on AI Agents: Capabilities and Safety @COLM_conf
This workshop explores AI agents’ capabilities—including reasoning and planning, interaction and embodiment, and real-world applications—as well as critical safety challenges related to reliability, ethics, and human-agent interaction.
Non-archival submission deadline: June 23, 2025!
🌐Website: sites.google.com/view/aia-wo…
📜OpenReview: openreview.net/group?id=colm…
👥Program committee/Reviewer sign-up form: forms.gle/5vHzyGxjUgSMNKMXA
✉️Mailing list: [email protected]
📅WikiCFP: wikicfp.com/cfp/servlet/even…
Shout out to our amazing organizing team members @Zhou_Yu_AI @ysu_nlp @uiuc_aisecure @ Yi Zhang @ Baolin Peng @ Jim Zhiwei Liu @DanRothNLP @LyleUngar @sharathguntuku @yugu_nlp @xy2437 @jeffrey_ch0 @AnnieFeng6 @Haoyu_Wang_97 @ZRChen_AISafety @raphaelshu @yooli23
Thanks @MSFTResearch for all the support! Excited to share that our work is also accepted by #ICLR2025!
- Paper: arxiv.org/abs/2410.02052
- Code: github.com/microsoft/ExACT
- Website: agent-e3.github.io/ExACT/
ExACT combines Reflective-MCTS and Exploratory Learning to improve AI agents' decision-making, enabling test-time compute scaling. Learn how these methods help agents refine strategies for state-of-the-art performance and improved computational efficiency: msft.it/6017Um7P3
Website link was broken… it’s now fixed!
To effectively solve modern computer tasks, AI agents need to be able to strategically explore the environment and efficiently learn from past interactions.
We present R-MCTS and Exploratory Learning for building o1-like models for agentic applications. Our GPT-4o powered R-MCTS agent creates SOTA performance on VisualWebArena. Notably, R-MCTS and Exploratory Learning (without MCTS) demonstrate the compute scaling properties in both training and testing time!
🌐: agent-e3.github.io/rmcts-exp…