As AI systems start doing research-level mathematics, how do we evaluate them, and how do we trust what they produce? Last week I gave this talk at the Penn AI Seminar on my research group's and @Rabdos_AI 's work on "scalable oversight" in the setting of mathematics: youtube.com/watch?v=n8ATwVq5… Scalable Oversight concerns techniques to monitor, evaluate, and control AI systems, even when they become more capable than the humans supervising them. Standard alignment methods like RLHF rely on humans directly judging model outputs. As AI moves into specialized or superhuman domains, human supervisors can no longer reliably judge correctness or safety. Without a reliable training signal, models risk reward hacking or developing unaligned behaviors that humans can't spot. That gap only widens as we move from RLHF toward recursive self-improvement. Mathematics is the ideal testbed. Correctness is, in principle, well defined, and we are already at the point where human checking can't keep up. This is also the principle behind Rabdos: can we create data that autonomously and continuously evaluates and improves AI systems? References: 1. MathDuels: A Self-Play Benchmark That Grows arxiv.org/abs/2604.21916 2. Do We Need Frontier Models to Verify Mathematical Proofs? arxiv.org/abs/2604.02450 3. Learning Strategies to Break Judges arxiv.org/abs/2609.33773 Joint work with PhD students Oscar Xu, Shibo Jin, Guruprerana Shabadi, Aaditya Naik, postdoc Shreya Arya, and colleague Rajeev Alur. #AI #Mathematics #LLM #AIEvaluation #ScalableOversight

I am at COLM'26 this week in San Francisco with my students as well as folks from @Rabdos_AI . We have three cool papers on timely and exciting topics; drop by if you are attending! > MathDuels: A Self-Play Benchmark That Grows arxiv.org/abs/2604.21916 Wednesday 11-1, Grand Ballroom #121 Leaderboard: mathduels.ai/ > Detecting Safety Violations Across Many Agent Traces arxiv.org/abs/2604.11806 Thursday 11-1, Grand Ballroom #129 > Do We Need Frontier Models to Verify Mathematical Proofs? arxiv.org/abs/2604.02450 Thursday 4:30-6:30, Imperial Ballroom #21

This is the first semester in my teaching career that I have been able to use LLMs to improve the learning experience. One way I’ve been doing this is simply feeding my static slides from each lecture into an agentic framework to create live demos in class. For my software analysis class (cis547.github.io/), I used Claude Code to demo different program correctness techniques, from random testing using AFL++ to theorem proving using Dafny. In the past, I spent hours just debugging the setup of these tools. Now Claude does it autonomously for me in *under 5 minutes*, giving me a standalone Docker container with a web UI. Check out this demo of an automated debugging technique called Delta Debugging that I covered this week: static.us.edusercontent.com/… As a bonus, you yourself learn subtle new things about the technical content. For me this happened with a concurrency fuzzing technique called PCT, where Claude showed how P(bug per run) is higher but rate of bugs per second is lower if you put threads to “sleep” versus simply preempt them. It is an even bigger privilege to be teaching in these times. LLMs can now do almost anything so it all comes down to instructor taste and judgment.

Every year I take an international trip, I measure in-flight internet connectivity as an indicator of global technological progress. 2026 represents a *huge* step up in this measure, entirely single-handedly due to @Starlink. They have essentially solved this problem from 0 to 1.

Incredibly proud of PhDs #8, #9, and #10 -- congratulations Drs. Jiani Huang (@jiani_huang_ai), Aaditya Naik (@aaditya_naik), and Adam Stein (@adamlsteinl)! I am lucky and grateful for the privilege of working with the three of you!

Friends, followers, and strangers on X, I recently got excited about mapping the jagged frontier of AI models in math and other STEM areas. With my colleague Prof. Rob Ghrist @prof_g , I founded Rabdos AI rabdos.ai to create original, research-level problems at scale to advance frontier AI capabilities. In the coming weeks, we'll be sharing exciting technical updates through @Rabdos_AI. Please give us a follow if you're interested in model evaluation, AI for math/science, autoformalization in Lean, robustness of world models, and much more.

Penn-goin! Elated that my amazing daughter Isha was admitted to Penn in the Mechanical Engineering program's Class of 2030! Lots of gratitude to her teachers, our family and friends, my colleagues, and even my students who helped in any way possible. We as a family got to experience the stressful process of US college admissions firsthand. Sending good wishes to others going through it right now. A reminder that admission decisions are never a measure of your worth as a person. thedp.com/article/2025/12/pe… Isha's website with artwork and writings: ishanaikfineart.com/ #PennEngineeringProud @MEAM_Penn #classof2030

It is easy to overlook but hard to overstate how big a role "test-time inference" has played in modern LLM gains in reasoning. I myself wasn't sure, so I put GPT-5 and Gemini-2.5 to a classic programming puzzle called variable shadowing (en.wikipedia.org/wiki/Variab…): Both models in "Thinking" mode solved it correctly but took around 2 minutes. The thinking was elaborate (and correct): "I'm currently focused on rewriting the user's C-like code, aiming to replace all instances of 'y' with 'x'. However, I've hit a snag. The crucial part is to avoid this substitution in specific scenarios. " More surprisingly, both models in "Instant" mode got it consistently wrong, giving a glimpse into their pre-training levels of reasoning which seem to have hardly improved over the years if at all. Is there a middle ground which reliably answers this simple question correctly without thinking for too long? The best solution today remains prompt engineering. Appending "Think step by step before answering" to the prompt causes both models to engage in a brief monologue which turns out to be crucial to subsequently generating the correct program. The recent "Auto" mode of LLMs is intended to automatically strike this balance and forgo the need for prompt engineering. But it currently errs on the side of over-thinking, or worse, under-thinking. LLMs will undoubtedly get better at calibrating their "Auto"mode but I am equally excited about emerging architectures like diffusion models, which are not constrained by left-to-right generation and can iteratively refine their answer the way humans typically do.