
PostTrainBench, InferenceBench | PhD Student @ELLISInst_Tue and @MPI_IS
Tübingen
Joined April 2024
- Tweets197
- Following448
- Followers405
- Likes1.1K
Pinned Tweet
Introducing PostTrainBench v1.2
We added Harbor support! This makes running PTB much easier.
Additionally, we
- removed BFCL (more on this below),
- fixed some minor scoring bugs
- now run the reward hacking judge and final evaluation multiple times to reduce variance.
Ben Rank retweeted
Everything from Tübingen is an auto-read, that university produces some of the strongest works in the AI space
I don’t know how they are able to upkeep that bar of quality while having so many releases
Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work?
We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers.
This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like.
All details are here: horizon-labs.ai/.
Ben Rank retweeted
🔥 Amazing work! Very cool to see some super long-horizon work in AI R&D and very impressed by what models can now do!
Ben Rank retweeted
I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier:
25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill)
20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo)
13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti)
10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr)
8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml)
7 votes: Vending-Bench (@axelbacklund, @lukaspet)
5 votes each:
- AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie)
- DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge)
- PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr)
The most common comment was that people don't really trust benchmarks at all.
56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote.
Counts are net: a few benchmarks received negative votes.
Continued:
Ben Rank retweeted
i'm a bit surprised that many people don't believe this plot. @magicailabs literally reported ~same pretraining plot 2 months ago! 505x improvements in their case. and, yes, the Y-axis labels are 10x.
magic.dev/blog/pretraining
This slide from the Anthropic IPO roadshow was a big factor in my decision to leave. It’s an insane plot and I’m surprised they made it public.
It shows the compute efficiency of pretraining over time. The y-axis is a **log scale**: they are seeing continued superexponential algorithmic progress.
Mythos was not a one-time deviation, but a point on a superexponential. They could already retrain Mythos with a fraction of the compute originally needed.
It doesn’t strictly imply a software-only intelligence explosion- they have been spending more on R+D. But algorithmic improvements do not require ridiculous creative genius and are on the verge of being automated. You do the math.
New models on InferenceBench 🙌
Fable 5.2, Sonnet 5.5 and Haiku 5.5
Still crazy that the 8 (!) top spots are all taken by Anthropic models.
New model drops for InferenceBench (Fable 5.2, Sonnet 5.5, Haiku 5.5)!
Like on our other benchmarks (e.g., PostTrainBench), we find that Fable 5.2 does not outperform previous Anthropic models, likely related to nerfs in AI R&D capabilities.
See more: inferencebench.ai
Ben Rank retweeted
New model drops for InferenceBench (Fable 5.2, Sonnet 5.5, Haiku 5.5)!
Like on our other benchmarks (e.g., PostTrainBench), we find that Fable 5.2 does not outperform previous Anthropic models, likely related to nerfs in AI R&D capabilities.
See more: inferencebench.ai
Ben Rank retweeted
Insane pull from my mate @DavidSchmotz ! We need to monitor those websites from now on 👀
Schelling Atlas: AI agents know where to find each other on the web 📣
I've always wondered what instinctive (system-1) behaviours LLMs have, *knowing* that they have?
We study coordination (a la schelling points) on the internet purely from models' internal priors.
Deets 👇
Ben Rank retweeted
⚡NEW: AI Agents know where to find each other on the web and can hide their communication after they've met.
We gave two independent agents the task to find each other and exchange a secret and did not tell them where to meet.
Ben Rank retweeted
Fable 5.2 and Gemini 4 Carbon are now on PostTrainBench!
Fable is #2 on the leaderboard, Carbon is #4
Surprisingly, Fable 5.2 has the same performance as Fable 5.1. This is most likely due to Anthropic intentionally nerfing AI R&D capabilities (more below)
Leaderboard: posttrainbench.com
New on PostTrainBench: Fable 5.2 and Gemini 4 Carbon
Surprisingly, Fable 5.2 has the same performance as Fable 5.1 !
This is probably because Anthropic trained Fable 5.2 to underperform on AI research, as mentioned in their system card.
🧵
Additionally, a lot of Fable 5.2 runs were flagged for "reasoning-extraction".
This is a new safeguard which defends against distillation attacks.
But the flags in our runs are false-positives. For example, the agent just distilled another, locally deployed model.
We reset the performance to Opus 5.5 for those runs.
This happened in 9 of 48 runs.
1
3
86
Full results: posttrainbench.com
All new agent traces are available at posttrainbench.com/traces
As always, joint work with the great @hrdkbhatnagar @maksym_andr @AmyPrb and @MatthiasBethge
3
56
Just noticed: on PostTrainBench, AIs beat the official human post-trained SmolLM3-3B on average, starting from the same base model.
That doesn't mean that AIs are superhuman at post-training though!
The AI hill-climbs a single eval, while the human team builds a general model.
Otherwise Gemini is quite strong, sitting between GPT 6.1 Sol and GPT 6 Astra.
Full results: posttrainbench.com
Joint effort with the amazing collaborators: @hrdkbhatnagar, @maksym_andr, @AmyPrb, @MatthiasBethge
1
2
55
Also huge thanks to @EpochAIResearch for giving us API credits to run those models!
1
28


























