@regunivers
🇺🇸Account based in
About this account
- Account based in
- United States
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
On the job market | PhD student @CarnegieMellon
Joined January 2022
- Tweets90
- Following129
- Followers256
- Likes355
Zeyu Zheng retweeted
Tinker's new StableReinforce objective is Reinforce + Score Centering. Here's how I reverse-engineered it 🧵
Off-policy RL is difficult; runs can suddenly collapse because of async RL staleness or numerical mismatch.
StableReinforce is a new method on Tinker that keeps your RL runs stable, even when learning drifts from sampling.
tinker-docs.thinkingmachines…
Zeyu Zheng retweeted
Three days before I was pushed out of OpenAI I posted a Formal Verification Manifesto in a core research Slack channel. It listed OpenAI systems that could benefit from formal verification.
The only person who responded was @sama, who was very supportive. Not sure what the moral of this story is, but it's never too late for OpenAI to invest seriously in software & hardware formal verification!
On a more serious note, @theoremlabs and @AnthropicAI did something amazing. They showed how to verify software at scale without requiring developers to write specifications in Rocq or Lean. Instead, they used existing fuzz harnesses as specifications and LLMs to generate machine-checked proofs.
Of course, fuzz harnesses specify only some correctness properties. In the simplest case, the proofs establish crash-safety, not full functional correctness.
We need community-maintained open-source projects with reusable formal specifications and correctness proofs for major libraries, kernels, compilers, virtual machines and hardware.
Human review of critical specifications and LLM-generated patches is essential. A model could satisfy a specification simply by removing functionality.
As AI writes and rewrites critical infrastructure, formal verification should become a key part of AI control. Critical properties of AI-generated code should be machine-checked rather than taken on faith.
AI is making vulnerabilities easier to find and exploit. Frontier LLMs and reusable formal specifications could finally make formal verification practical and economical at scale, while keeping it understandable to humans.
Kudos to Theorem co-founders @rajashree and @diagram_chaser!
New on our Frontier Red Team blog: using Claude to formally verify software at large scale.
Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time).
With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks.
Read our research in full here: anthropic.com/research/forma…
Zeyu Zheng retweeted
With Haotian's permission, I'm sharing this blog post on Twitter. Naturally, to protect his privacy, I won't tag or share his Twitter handle.
It's a really meaningful post. It discusses the issue in JPC of needing to subtract the mean of the scores under the behavior policy, and details some related experiments, including comparisons of top-k values and full vocabulary.
Reading this reminded me of a few papers from my time working on OPD, and it left me wondering about a couple of things:
1. Why do models as large as DeepSeek still use full vocabulary when doing OPD?
2. Why do many others, like Thinking Machine, which uses sampled tokens, take different approaches?
3. Meanwhile, Qwen uses top-k tokens, and while they mentioned in their experiments that the difference wasn't huge, it honestly feels like the gains are actually quite significant.
zhuanlan.zhihu.com/p/2096559…
Zeyu Zheng retweeted
AI math needs two scoreboards: proved and disproved.
In reuF2F, Gemini 4 Argon resolved 109 of 244 formalized questions within two attempts: 61 proofs, 48 refutations.
Nearly half the progress was showing the conjecture was false.
xitter.cf/regunivers/status/2112…
ALT Stacked horizontal bars showing author-reported reuF2F results after two attempts per problem, on 244 formalized undergraduate research questions under the same Leanimum-agent harness. Gemini 4 Argon high effort: 61 proved, 48 refuted, 135 unresolved. Claude Opus 5.5 max: 57 proved, 50 refuted, 137 unresolved. GPT-6.1 Sol max: 47 proved, 53 refuted, 144 unresolved. GPT-6 Astra max: 42 proved, 50 refuted, 152 unresolved. Proof and refutation are both valid resolutions. Different models use different token budgets. Up to six hours per attempt. Table 5 of Zheng et al. reuF2F preprint, reviewed Oct 21, 2026; not independently rerun. Counts reconstructed from published rounded percentages.
1/5
Verification has become the bottleneck as LLMs start proving conjectures at scale! No worries, Lean can help. But how well can AI agents solve open problems formalized in Lean, end to end?
Introducing reuF2F: 244 open math problems from Research Experiences for Undergraduates (REU) programs, spanning a broad range of math areas, formalized by experts in Lean.
We also build Leanimum-agent, a minimal coding agent for Lean, so every model is compared under the same harness.
Even the best-performing model resolves only 109 out of 244 problems. More below 🧵
Zeyu Zheng retweeted
Today, we release our first model, Flint-40B-A2B, a fast local coding model built with our automated research system. huggingface.co/RecursiveSI/f…
1. Strong performance. 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, and 74.2% on SWE-bench Verified.
2. It is trained from scratch (tokenizer->Pretrain->Midtrain->SFT->RL).
3. Fast in your local laptop. Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark.
4. We build the complete model training pipeline in 4 month!
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
Zeyu Zheng retweeted
Also check out mini-Lean-agent, another recent minimal agent harness for Lean by @tomas__ortega et al!
arxiv.org/abs/2610.17331
Now it's on arXiv!
arxiv.org/abs/2610.27046
1/5
Verification has become the bottleneck as LLMs start proving conjectures at scale! No worries, Lean can help. But how well can AI agents solve open problems formalized in Lean, end to end?
Introducing reuF2F: 244 open math problems from Research Experiences for Undergraduates (REU) programs, spanning a broad range of math areas, formalized by experts in Lean.
We also build Leanimum-agent, a minimal coding agent for Lean, so every model is compared under the same harness.
Even the best-performing model resolves only 109 out of 244 problems. More below 🧵
Zeyu Zheng retweeted
New benchmark: reuF2F
244 open math problems from Research Experiences for Undergraduates (REU) programs, formalized in Lean.
Agents are given the Lean statement of an open problem, and produce formal proofs that are checked by Lean.
Paper: zeyu-zheng.github.io/reuF2F.… (1/n)
1/5
Verification has become the bottleneck as LLMs start proving conjectures at scale! No worries, Lean can help. But how well can AI agents solve open problems formalized in Lean, end to end?
Introducing reuF2F: 244 open math problems from Research Experiences for Undergraduates (REU) programs, spanning a broad range of math areas, formalized by experts in Lean.
We also build Leanimum-agent, a minimal coding agent for Lean, so every model is compared under the same harness.
Even the best-performing model resolves only 109 out of 244 problems. More below 🧵
1/5
Verification has become the bottleneck as LLMs start proving conjectures at scale! No worries, Lean can help. But how well can AI agents solve open problems formalized in Lean, end to end?
Introducing reuF2F: 244 open math problems from Research Experiences for Undergraduates (REU) programs, spanning a broad range of math areas, formalized by experts in Lean.
We also build Leanimum-agent, a minimal coding agent for Lean, so every model is compared under the same harness.
Even the best-performing model resolves only 109 out of 244 problems. More below 🧵
4/5
How do we grade an open conjecture? It might be false! Each task includes the conjecture AND its full negation, and the agent can choose to prove either.
Each submission is a git patch, replayed in a fresh evaluation container and checked by Comparator with only the standard axioms allowed. It must pass both the Lean kernel and nanoda, an independent kernel implementation.
Correctness of formalization has always a concern for any Lean benchmark. What if a problem is misformalized?
We guard against it in two ways:
• Every problem is formalized by Lean experts, cross-checked by frontier models, and given a final review by Mathlib contributors on our team.
• Even if a formalized statement P drifts from the authors' intent, two-way grading ensures that a verified proof of P or ¬P still provides valid evaluation signal.
5/5
Pass@2 with the same Leanimum-agent harness and a generous budget:
Gemini 4 Argon (high) 44.67% · Claude Opus 5.5 (max) 43.85% · GPT-6.1 Sol (max) 40.98% · GPT-6 Astra (max) 37.70% · GPT-6 Sol (max) 19.26% · Kimi K3 9.84% · Qwen3.8-Max 9.02% · Gemini 3.8 Flash 7.38%.
Some take aways:
More thinking can beat more attempts! Increasing reasoning effort consistently improves performance, and can be more token-efficient than repetition. For example, a single high-effort Opus 5.5 attempt beats two medium-effort attempts (29.30% vs. 28.28% pass rate), while using fewer output tokens (62.81K vs. 81.35K). From low to max effort, pass@1 rises from 9.22% to 42.01% for Opus 5.5 and from 1.43% to 39.55% for GPT-6.1 Sol !!!
Cost matters in real life! The dashed line demonstrates the cost/performance Pareto Frontier, which consists mostly of GPT-6.1 Sol and Opus 5.5 effort levels, with Gemini 4 Argon reaching the highest accuracy. Open source frontiers Kimi K3 and Qwen3.8-Max spend the most, yet both remain below 10% pass@2.
Zeyu Zheng retweeted
New paper led by @seungonekim:
"Into the Unknown: Benchmarking AI agents on open scientific research problems"
arxiv.org/abs/2610.19159
Given a paper and research direction, agents carry out research and write a follow-up paper.
We worked with scientists from US national labs to create follow-up directions and evaluation rubrics, from areas such as plasma physics and materials science.
None of the agents that we tested produced a paper that passed all of the rubric items. It will be exciting to see future systems that tackle these kinds of tasks!
🧪Can AI agents take a scientific paper and write a meaningful follow-up to it?
We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs:
@LosAlamosNatLab @SandiaLabs @BerkeleyLab @AmesNatLab @PPPLab @Livermore_Lab
Zeyu Zheng retweeted
Can AI agents perform open-ended research beyond well-scoped optimization?
We curated 15 open problems across scientific domains, with automated evaluation using expert rubrics.
Many agents failed integrity checks; none, including GPT-6, fully met the experts’ bar.
🧪Can AI agents take a scientific paper and write a meaningful follow-up to it?
We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs:
@LosAlamosNatLab @SandiaLabs @BerkeleyLab @AmesNatLab @PPPLab @Livermore_Lab
Zeyu Zheng retweeted
Gym-Anything won Best Paper at the COLM Lifelong Agents Workshop!
Congrats @PranjalAggarw16!!
Zeyu Zheng retweeted
I will be presenting Gym-Anything oral at #COLM2026 workshop on lifelong learning agents.
🗓 12pm at Imperial Ballroom 6
If you are interested in computer use agents or how to make rl environments, do check it out!
Zeyu Zheng retweeted
I'm not at #COLM2026, but three PhD students from the L3 Lab are there!
- Pranjal Aggarwal (@PranjalAggarw16)
- Weihua Du (@StigLidu)
- Tianran Sun (@JOYSUN200402)
Say hello and check out the L3 lab papers: cmu-l3.github.io/colm-2026/
Zeyu Zheng retweeted
Reading all those recent beautiful essays by Terry Tao, Kevin Buzzard and @wtgowers, I could not resist, but write up my deeply personal perspective on how I perceive and see things:
docs.google.com/document/d/e…
Zeyu Zheng retweeted
Computer-use agents need to be FAST
cua-speedrun lets you measure how fast they actually are
No single model family is optimal for speed, cost, and task performance, and no open-weight model is on the frontier
Amazing work led by @PranjalAggarw16, @JangLawrenceK, @kohjingyu!
Computer-use agents are extremely good!
But they need to improve on a metric which most benchmarks don't track. How much time do CUAs take to complete a task?
Introducing cua-speedrun: a benchmark for accuracy, speed, and cost under matched infra.
cuaspeedrun.com
👇🧵