Assistant Professor at CMU. Marathoner, @thesisreview.
Joined April 2011
- Tweets1.5K
- Following241
- Followers7.7K
- Likes2.6K
Sean Welleck retweeted
That’s a great benchmark to look at
1/5
Verification has become the bottleneck as LLMs start proving conjectures at scale! No worries, Lean can help. But how well can AI agents solve open problems formalized in Lean, end to end?
Introducing reuF2F: 244 open math problems from Research Experiences for Undergraduates (REU) programs, spanning a broad range of math areas, formalized by experts in Lean.
We also build Leanimum-agent, a minimal coding agent for Lean, so every model is compared under the same harness.
Even the best-performing model resolves only 109 out of 244 problems. More below 🧵
Sean Welleck retweeted
If you've noticed a distinct dialect emerging in frontier AI models and struggled to understand what they're talking about, you're not crazy!
We turned to psycholinguistics to investigate what makes the writing style so weird, and why this might be happening
🚨New preprint! 🧵
Sean Welleck retweeted
AI agents can't yet carry out open scientific research
Into the Unknown benchmarks 9 frontier agents on 15 unsolved research directions curated by DOE scientists. None of the 135 submissions meets the scientist's bar, and 114 fail an integrity check for fabricated or inconsistent results.
Sean Welleck retweeted
REUF2F: Benchmarking coding agents on formalized undergraduate research problems. ~ Zeyu Zheng et al. zeyu-zheng.github.io/reuF2F.…
#AI4Math
Sean Welleck retweeted
Happy to see mini-Lean-agent style harnesses becoming popular!
Systems that require complex lean search tools and other scaffolding might be overkill nowadays.
Modern models can perform very well with just access to bash!
Also check out mini-Lean-agent, another recent minimal agent harness for Lean by @tomas__ortega et al!
arxiv.org/abs/2610.17331
Also check out mini-Lean-agent, another recent minimal agent harness for Lean by @tomas__ortega et al!
arxiv.org/abs/2610.17331
reuF2F is now on arxiv 🎉
arxiv.org/abs/2610.27046
New benchmark: reuF2F
244 open math problems from Research Experiences for Undergraduates (REU) programs, formalized in Lean.
Agents are given the Lean statement of an open problem, and produce formal proofs that are checked by Lean.
Paper: zeyu-zheng.github.io/reuF2F.… (1/n)
Sean Welleck retweeted
Now it's on arXiv!
arxiv.org/abs/2610.27046
1/5
Verification has become the bottleneck as LLMs start proving conjectures at scale! No worries, Lean can help. But how well can AI agents solve open problems formalized in Lean, end to end?
Introducing reuF2F: 244 open math problems from Research Experiences for Undergraduates (REU) programs, spanning a broad range of math areas, formalized by experts in Lean.
We also build Leanimum-agent, a minimal coding agent for Lean, so every model is compared under the same harness.
Even the best-performing model resolves only 109 out of 244 problems. More below 🧵
Sean Welleck retweeted
What is TMLR doing under a 300% increase in submissions?
- Suspensions of >150 authors
- Publicly naming major violations
- Submission quotas distinguishing slop patterns
- Desk rejections 6% → 53%; focus on clarity
- AI reviews evaluating soundness
blog.tmlr.org/2026/adapting-…
Amazing work and collaborative effort led by Zeyu Zheng (@regunivers)!
Paper: zeyu-zheng.github.io/reuF2F.…
Benchmark: github.com/cmu-l3/reuF2F
Leanimum Agent: github.com/cmu-l3/Leanimum-a… (7/7)
New benchmark: reuF2F
244 open math problems from Research Experiences for Undergraduates (REU) programs, formalized in Lean.
Agents are given the Lean statement of an open problem, and produce formal proofs that are checked by Lean.
Paper: zeyu-zheng.github.io/reuF2F.… (1/n)
1/5
Verification has become the bottleneck as LLMs start proving conjectures at scale! No worries, Lean can help. But how well can AI agents solve open problems formalized in Lean, end to end?
Introducing reuF2F: 244 open math problems from Research Experiences for Undergraduates (REU) programs, spanning a broad range of math areas, formalized by experts in Lean.
We also build Leanimum-agent, a minimal coding agent for Lean, so every model is compared under the same harness.
Even the best-performing model resolves only 109 out of 244 problems. More below 🧵
Sean Welleck retweeted
1/5
Verification has become the bottleneck as LLMs start proving conjectures at scale! No worries, Lean can help. But how well can AI agents solve open problems formalized in Lean, end to end?
Introducing reuF2F: 244 open math problems from Research Experiences for Undergraduates (REU) programs, spanning a broad range of math areas, formalized by experts in Lean.
We also build Leanimum-agent, a minimal coding agent for Lean, so every model is compared under the same harness.
Even the best-performing model resolves only 109 out of 244 problems. More below 🧵
New paper led by @seungonekim:
"Into the Unknown: Benchmarking AI agents on open scientific research problems"
arxiv.org/abs/2610.19159
Given a paper and research direction, agents carry out research and write a follow-up paper.
We worked with scientists from US national labs to create follow-up directions and evaluation rubrics, from areas such as plasma physics and materials science.
None of the agents that we tested produced a paper that passed all of the rubric items. It will be exciting to see future systems that tackle these kinds of tasks!
🧪Can AI agents take a scientific paper and write a meaningful follow-up to it?
We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs:
@LosAlamosNatLab @SandiaLabs @BerkeleyLab @AmesNatLab @PPPLab @Livermore_Lab
Sean Welleck retweeted
🧪Can AI agents take a scientific paper and write a meaningful follow-up to it?
We introduce Into the Unknown, an AI agent benchmark of 15 open scientific research problems. Each problem comes from a scientist currently or formerly at US DOE National Labs:
@LosAlamosNatLab @SandiaLabs @BerkeleyLab @AmesNatLab @PPPLab @Livermore_Lab
