Rujun Han (@HanRujun) profile picture

Senior Research Scientist @Google working LLM agents, post-training and distillation. NLP+ML Ph.D. from @USC_ISI. Ex-@AWS AI, ex-@federalreserve.

Sunnyvale, CA
Joined May 2019
Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)Photo by Rujun Han (@HanRujun)
Join the arena ๐Ÿ”ฅ letโ€™s build the best engineering agent
๐Ÿค– Made with AI
1
1
7
7,057
Rujun Han retweeted
New Google paper finds that the best AI agent scores only 50.6% on physical engineering jobs, often by missing a hard safety or feasibility rule. Engineering benchmarks for LLMs mostly test textbook questions or toy problems that top agents already handle well. Real engineers work from piles of PDFs, spreadsheets, and CAD files and must issue designs that meet code and stay cheap. They built PLE, 85 electrical, civil, structural, and mechanical jobs written or checked by practising engineers. Agents must read about 61 project files per task and deliver schedules, reports, or 2D and 3D drawings. Claude Opus 5.5 in Claude Code topped 14 models at 50.6%, using 27M tokens and 63 minutes per task. On design tasks its outputs met 91% of constraints, but only 41.3% met every hard constraint. If you build engineering agents, gate every design on hard safety and code limits, and spend effort on model choice before tuning the agent scaffold.
23
13
8
109
8,501
Rujun Han retweeted
Banger paper from Google Cloud AI Research on agents doing professional engineering work. (bookmark it) For professional engineering deliverables, the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task. They introduce a benchmark called PLE (Physical-Engineering's Last Exam). It has 85 tasks across electrical, civil, structural, and mechanical engineering. Solving a task means producing a billable deliverable, such as a design package a licensed reviewer would sign off on, from a multi-format project corpus with physics and building-code requirements. Some tasks are written end to end by practicing engineers. Others start from an engineer's problem statement, and an adversarial Creator-Solver agent pipeline builds the project environment, grader, and reference deliverables before experts validate them and make them harder. Frontier models were tested under seven public agent harnesses. Most existing engineering benchmarks are textbook QA or toy design problems that leading agents already saturate. arxiv.org/abs/2610.30537 Chat with Paper: academy.dair.ai/papers/ple-pโ€ฆ
12
23
2
124
8,722
Rujun Han retweeted
Banger paper from Google. It's standard practice to sample several agent rollouts and trust the answers they agree on. This Google paper shows that agreement can hide shared errors, while disagreement often points to the correct alternative. VeriHarness turns the same base model into an agentic verifier with two jobs. One resolves claims where rollouts disagree by checking workspace evidence. The other challenges claims that every rollout agrees on and looks for requirements they all missed. Across five long-horizon benchmarks, it gives the best selection scores among the baselines tested. With evidence-backed revision, it adds 6.2 points over a single rollout with Gemini 3.5 Flash and 6.4 points with Claude Opus 4.8. The authors also release about 26,000 rollouts. Paper: arxiv.org/abs/2610.00972 Chat with Paper: academy.dair.ai/papers/verihโ€ฆ
55
68
7
446
31,905
Rujun Han retweeted
New Google Paper says financial deep-research agents are far better at reconstructing the past than anticipating what comes next. Across 17 baselines plus FinanceHarness, every model they tested stayed below 40% overall on 400 expert-annotated questions. The harness matters: with the same Qwen3.6-27B backbone, moving from a simple search loop to the full finance-oriented tool and workflow stack raised the overall score from 25.3% to 32.4%. Extra training barely changed that result, adding only 0.4 percentage points after Group Relative Policy Optimization. So the main bottleneck is no longer just search, citation, or report structure. Financial research agents need better causal and scenario reasoning, because a cleaner evidence pipeline does not automatically produce better forward-looking judgment. โ€“ arxiv. org/abs/2607.27853 Title: "FinanceHarness: Autonomous Financial Deep Research Framework"
12
28
7
172
12,477
Rujun Han retweeted
If you are into OPD (on-policy distillation), please don't forget to check our Speculative OPD (Speculative Knowledge Distillation). Idea: For each token, sample it from the student but discard if it teacher is unlikely to generate it (top-K), and resample from teacher. This dynamically switches between on-policy and supervised distillation.
6
20
2
131
11,910
Rujun Han retweeted
๐Ÿงฉ How can we assign fine-grained credit over long tool-use trajectories and let agents learn from past attempts in agentic reinforcement learning when rewards are no longer verifiable? Excited to share RubricEM, an RL framework for long-form deep research agents that plan, search, use tools, and write reports without exact answer checks. ๐Ÿ“– Paper: arxiv.org/abs/2605.10899 (1/n)
3
27
4
112
17,192
We are launching two powerful updates to Deep Research in the Gemini API, now with better quality, MCP support, and native chart/infographics generation. Use Deep Research when you want speed and efficiency, and use Max when you want the highest quality context gathering & synthesis using extended test-time compute โ€” achieving 93.3% on DeepSearchQA and 54.6% on HLE.
230
429
62
4,938
417,046
Rujun Han retweeted
Supervised RL accepted to ICLR 2026! Thanks to all my co-authors for the great work and especially @IHung_Hsu :)
guess it's time to wrap up 2025 :) - OpenVLThinker: iterative SFT-RL cycles arxiv.org/abs/2503.17352 - Supervised RL: step-wise rewards via seq similarity w/ expert traces arxiv.org/abs/2510.25992 - Joined our amazing multimodal team @xAI ๐Ÿš€ And earlier this year: - DuoGuard: generator-classifier setting for synthetic data generation arxiv.org/abs/2502.05163
3
14
278
26,880
๐ŸงGiving your agent unlimited tool calls doesn't make it smarter. ๐Ÿ’กWhy? It lacks 'Budget Awareness'! Introducing Budget Tracker, a simple plug-in that enables more effective scaling behaviors: higher performance, lower cost. Paper: arxiv.org/pdf/2511.17006
1
17
3
31
6,806
๐Ÿง ๐Ÿš€ Excited to introduce Supervised Reinforcement Learningโ€”a framework that leverages expert trajectories to teach small LMs how to reason through hard problems without losing their minds. ๐Ÿคฏ Better than SFT && RLVR. Read more: huggingface.co/papers/2510.2โ€ฆ #llms #RL #reasoning
12
62
2
328
20,756
New research introduces the Test-Time Diffusion Deep Researcher (TTD-DR) framework, advancing AI agents for complex research: goo.gle/4pwtplf
TTD-DR is a Deep Researcher agent that models research report writing as a diffusion process, where a first draft is gradually polished into a high-quality final version. Read about its impressive results in long-form report writing and complex reasoning tasks: goo.gle/4ncW5yp
4
5
1,045
Rujun Han retweeted
Excited to share our #ACL2025NLP paper, "๐‚๐ข๐ญ๐ž๐„๐ฏ๐š๐ฅ: ๐๐ซ๐ข๐ง๐œ๐ข๐ฉ๐ฅ๐ž-๐ƒ๐ซ๐ข๐ฏ๐ž๐ง ๐‚๐ข๐ญ๐š๐ญ๐ข๐จ๐ง ๐„๐ฏ๐š๐ฅ๐ฎ๐š๐ญ๐ข๐จ๐ง ๐Ÿ๐จ๐ซ ๐’๐จ๐ฎ๐ซ๐œ๐ž ๐€๐ญ๐ญ๐ซ๐ข๐›๐ฎ๐ญ๐ข๐จ๐ง"! ๐Ÿ“œ If youโ€™re working on RAG, Deep Research and Trustworthy AI, this is for you. Why? Citation quality is critical for trust, but current metrics are falling short. Letโ€™s fix that! ๐Ÿงต [1/10]
2
9
1
46
4,569
Can many-shot ICL be cached and still tailored per test sample? We make it possible. ๐Ÿ’ก Excited to share that our paper, "Towards Compute-Optimal Many-Shot In-Context Learning," has been accepted to @COLM_conf! Paper: arxiv.org/pdf/2507.16217 #COLM2025 #LLMs #AI #ICL
1
1
3
493
Rujun Han retweeted
Thrilled to introduce "๐——๐—ฒ๐—ฒ๐—ฝ ๐—ฅ๐—ฒ๐˜€๐—ฒ๐—ฎ๐—ฟ๐—ฐ๐—ต๐—ฒ๐—ฟ ๐˜„๐—ถ๐˜๐—ต ๐—ง๐—ฒ๐˜€๐˜-๐—ง๐—ถ๐—บ๐—ฒ ๐——๐—ถ๐—ณ๐—ณ๐˜‚๐˜€๐—ถ๐—ผ๐—ป," a new deep research agent designed to mimic the iterative nature of human research, complete with cycles of planning, drafting, and revision. ๐Ÿš€๐Ÿš€ arxiv.org/pdf/2507.16075
12
84
8
448
37,120
I will be presenting โœจReverse Thinking Makes LLMs Stronger Reasonersโœจat #NAACL2025 ! We show that LLM can also benefit from reverse thinking -- a technique we often use to reason from a problem to a solution: - Improvements across 12 datasets - Outperforms SFT with 10x more data - Strong generalization to OOD datasets ๐Ÿ“… 4/30 2:00-3:30 Hall 3 Let's chat about LLM reasoning and its future directions!
๐Ÿšจ Reverse Thinking Makes LLMs Stronger Reasoners We can often reason from a problem to a solution and also in reverse to enhance our overall reasoning. RevThink shows that LLMs can also benefit from reverse thinking ๐Ÿ‘‰ 13.53% gains + sample efficiency + strong generalization! -- We train a student LLM to generate forward reasoning and backward question from a question, and backward reasoning from backward question using a joint objective. -- Across 12 datasets on commonsense, math, logical reasoning and NLI, RevThink shows an average 13.53% improvement over the student LLMโ€™s zero-shot performance and a 6.84% improvement over the strongest knowledge distillation baselines. -- Using only 10% of the correct forward reasoning from the training data, RevThink outperforms a standard fine-tuning method trained on 10x more forward reasoning. -- RevThink also exhibits strong generalization to 4 out-of-distribution held-out datasets. ๐Ÿงต
2
37
126
9,660
Happy to share that RevThink has been accepted to #NAACL2025 main conference! ๐ŸŽ‰We also release the code and data ๐Ÿ‘‡๐Ÿงต RevThink shows that LLMs can also benefit from reverse thinking (like we often do) ๐Ÿ‘‰13.53% gains on 12 datasets (including MATH, ARC, ANLI, etc) + sample efficiency + strong generalization!
๐Ÿšจ Reverse Thinking Makes LLMs Stronger Reasoners We can often reason from a problem to a solution and also in reverse to enhance our overall reasoning. RevThink shows that LLMs can also benefit from reverse thinking ๐Ÿ‘‰ 13.53% gains + sample efficiency + strong generalization! -- We train a student LLM to generate forward reasoning and backward question from a question, and backward reasoning from backward question using a joint objective. -- Across 12 datasets on commonsense, math, logical reasoning and NLI, RevThink shows an average 13.53% improvement over the student LLMโ€™s zero-shot performance and a 6.84% improvement over the strongest knowledge distillation baselines. -- Using only 10% of the correct forward reasoning from the training data, RevThink outperforms a standard fine-tuning method trained on 10x more forward reasoning. -- RevThink also exhibits strong generalization to 4 out-of-distribution held-out datasets. ๐Ÿงต
3
44
1
116
9,261
This Thanksgiving, Iโ€™m deeply grateful for opportunity to run for NAACL Board Member alongside so many incredible candidates who share a passion for making a difference! I hope to earn your support as we work together to shape the future of our community! #NLProc #Gratitude
3
19
4
115
22,409