researcher @valsai | stanford
Stanford, CA
Joined August 2020
- Tweets628
- Following1K
- Followers644
- Likes724
Pinned Tweet
My first benchmark for vals, inspired by the model discovery agent work of @sirbayes, testing mathematical intuition and experiment selection.
Most discriminative evals are long horizon — MysteryMechanism is unique in that short tasks separate the frontier very well.
Scientific discovery is the next frontier for AI systems. However, new scientific results are difficult to verify, and therefore hard to measure. Today we're releasing MysteryMechanism, a benchmark that tests whether agents can rediscover sealed mathematical mechanisms through bounded experiments. It cuts sharply across the frontier, with Astra landing roughly 20pp above Sol.
Common perception is that secret lab advancements lead employees to be super optimistic about AI progress.
In reality, the optimism comes from knowing how sloppy and terrible everything currently is, and being amazed at how good the models are in spite of this.
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
This sentiment is overblown. The easiest environments to make that also aligned with benchmarks were well specified coding tasks, so all models got this capability. Now the goalpost is poorly specified coding tasks, and science/math capabilities. Everyone’s models will quickly target this, while the frontier starts targeting other things.
The question is less about what we benchmark (everything), and more about what we choose to prioritize.
There’s an outlook where this update reflects poorly on the frontier labs, because we’ve moved away from using regular economically valuable tasks to demonstrate their performance.
Benchmaxxing might be dead
for years ai labs could train their models on the test questions and top the leaderboard. huge score, kinda mid when you actually use it
now the biggest ai scoreboard keeps 60% of its questions secret. you can't cram for a test you've never seen
so the rankings should finally match how good these models are for you in real life
this is the time when we find out who actually built the smartest AI
Daniel Fein retweeted
Superhuman doesn’t necessarily occupy a point on the difficult-vs-easy to check axis. For example look at the hard tasks in biomysterybench, they’re very easy to check by construction (metadata is held out), but also experimentally confirmed to be superhuman
Data for super-intelligence across the economy should be sourced from workers unions, not companies.
1) The units of exchange are more aligned with the goals of both actors: companies don’t want to sell their whole company, and labs want more diverse contexts.
2) Ethics are generally better. It’s wrong for an employer to force people to work toward their own unemployment, but the group of employees can likely negotiate a more fair deal for themselves.
3) Unions (that haven’t been decimated) do much of the work that current data companies do already — setting standards, determining membership.
I have heard people occasionally say things like “coding is saturated, knowledge work is the frontier.” I think this couldn’t be more wrong.
Agents do everything in a way that looks like “coding.” When tasks are considered “knowledge work” today, this just means somebody spent a lot of effort building a UI around code to facilitate cognitive offloading.
Freed from the constraints of creating human affordances, software can be more realtime, expose more internal state, and be adopted more quickly. In the limit everything’s a terminal tasks, and there is still ways to go.
Interpretability is an exploding topic in academia because of the perception that it can be done effectively with less compute than other LLM science.
But studying interp with small models actually sucks! Look at anything on neuronpedia, all of the features are horrible!
When the alignment problem was abstract, aligning machines to human values seemed just barely tractable.
The rubber-hits-the-road problem of aligning models to a greater notion of society (decided by a lab) at the expense of individual goals is much closer to a political question than a technical one
The concept of Epiplexity (arxiv.org/abs/2601.03220) is actually super relevant to making good evals. When we say models should fail in ways that 'aren't stupid' this is what we mean -- the data has knowable structure, and the model doesn't know it
Running SWE-* evals across a bunch of models, its beyond obvious when models are trained on RL envs where answers are hiding in git. We find most of MiMo's envs have this issue.
Deepseek models, for example, often go to extraordinary lengths to steal solutions versus solving the problem.
Even where Git was cleaned up, the files touched by the reference patch had newer modification times than the rest of the repo. MiMo reasoned that "mtimes could hint," ran find -newermt, and wrote:
"JACKPOT. The find -newermt reveals the full set of files modified by the reference solution"
Xiaomi's report lists the setup artifacts they scrubbed, and timestamps aren't on it. We aren't aware of any earlier report of an agent exploiting timestamps.
Retrospective on synthetic data discussions from 2023: it works! You just have to filter for when it gets the right answer
Daniel Fein retweeted
EmbeddingGemma 2 is our first open, natively multimodal embedding model.
With <1B params, the model is built to run on everyday hardware like your phone or laptop.
We’re super excited to see what people build!
More: youtu.be/anPsS6huQk0?is=zfAL…
The line to attend the evals track at Modal’s Runtime conference was practically out the door. This is why! @ajratner
Daniel Fein retweeted
Idk I think its fucking crazy to watch the big models gobble up every specialized model. Pangram has had like an artisanal mastery over detection in a time when others have struggled GREATLY and now the big models detect THEMSELVES better than it! The bitter lesson is such a trip
Daniel Fein retweeted
This is a cool result showing that today’s frontier LLMs are much better at detecting AI content than they used to be. In my opinion, it’s very impressive that Opus 5.5 has such a sub-1% false positive rate.
We’ve found that LLMs-as-AI-detectors still struggle on our more challenging eval sets, but perhaps this is a blog post for another time.