
maximum plasticity! @harvard & @tjhsst_official.
Joined September 2023
- Tweets133
- Following2.1K
- Followers865
- Likes3.3K
This is exactly the right approach! Big agents are much better at long-horizon physical tasks but aren't fast. Small policies are fast but can't reason and chain actions for long-horizon work. So add them together.
The industrial explosion is coming very soon...
We're announcing our first robot-use agent, Waddle-1. It controls bimanual robots on unseen tasks zero-shot.
Our agent is the most capable system evaluated by @robocurve, scoring 81 compared to 65 for GPT-6.2 Astra and 59 for Dream-1. It calls LLMs in the cloud to direct lower-level policies that run locally.
Waddle-1 is a first step towards agents that combine the capability of reasoning models and the efficiency of reactive policies. We expect such systems to achieve human-speed manipulation in the near future.
If you would like to deploy robots with our preview API, please tell us what you want to build here: waddlelabs.ai/api-access
5
527
OpenAI started reporting this benchmark in the GPT 5.6 model cards as an internal benchmark, so it's really cool to see the full results!
MedChemBench on Benchmark Almanac: benchmarkalmanac.com/benchma…
MedChemBench is a new benchmark for AI reasoning in medicinal chemistry.
It shows the progress our frontier models have made in their ability to help scientists use AI to identify promising drug candidates.
openai.com/index/introducing…
Brian Zhou retweeted
And as a result, the Tasmanian Government has published one one of the greatest Government Documents I have ever seen
Gemini 4 Neon now on the Benchmark Almanac: benchmarkalmanac.com/models/…
Introducing Gemini 4 Neon! Pushing the frontier across domains and punching above its model class in a meaningful way. To me, Neon feels like the right balance of intelligence, speed, and usability. Available now across @GoogleAIStudio, the API, and more!
3
312
Don't you ever get sick of Claude or ChatGPT making you the same cookie-cutter websites with Instrument Serif and Inter?
I asked Claude for "a font like Starry Night" and it drew one using WhateverFont, a tool that lets you make your own beautiful fonts: fonts.brianzhou.org
WhateverFont supports Greek and Cyrillic natively. For Arabic, Hindi, Japanese, Chinese and Korean, WhateverFont traces an open font of that script and draws it again with your font's pen.
Here's the snow font in six scripts:
1
1
56
Brian Zhou retweeted
Fable 5.2 didn't build this SF Muni bus backflip game, and it's smaller than the average webpage (1.4 MB)
I used Opus 5.5 and SWE-2 Flash instead! Everything you see is only made with code.
Brian Zhou retweeted
AI is getting better faster than our measurements are
I threw together SatBench, which measures how quickly major AI benchmarks become saturated (effectively beaten). Of 173 benchmarks cited by the Labs since 2023, 63% are now saturated. Saturation is happening faster each year.
Brian Zhou retweeted
Proud to announce the inclusion of Perceptron's recently released EgoHome fine-grained egocentric video understanding benchmark on Benchmark Almanac! Links below
I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there.
Website: benchmarkalmanac.com
Brian Zhou retweeted
This is super powerful. The world doesn't really have a good view of all the different models and how they compare to each other, since so many reports cherrypick which evals to run, and indexes like ECI or AAII are selected for various, sometimes opaque, constraints. Great work Brian!
I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there.
Website: benchmarkalmanac.com
You can also compare up to four models head to head on every benchmark they've all been evaluated on. Right now, Claude Fable 5.2 beats Gemini 4 carbon, GPT-6 astra and Muse Wonder on 21 of 45 benchmarks all four were evaluated on.
benchmarkalmanac.com/compare
The almanac is meant to stay current. Every morning, one job reads 26 sources (@latentspacepod ainews, @huggingface daily papers, @arxiv, @epochairesearch, lab model cards, eval hubs, bluesky…) for new benchmarks and models, and opens PRs for maintainers to review.
Please reach out if you would be interested in helping maintain this project!
This exists because I wanted an easy way to see:
a) what benchmarks and arenas already existed
b) how models perform on various aggregate evals
c) how benchmarks and evals have changed over time
Benchmark Almanac is fully open source. If you see a missing benchmark or score, want to log a known critique (leaked answers, a broken grader), or maintain a benchmark and want it to be featured, please join on as a contributor!
Contribute: github.com/brianlzhou/benchm…
Want to try your hand at making your own blend of benchmarks? Build your own index! You can add/drop evals and give your own custom weighting to each.
benchmarkalmanac.com/build
Evals also offer a window into a model's jagged capabilities. Anthropic leads 10 of the 13 contested bio and chem benchmarks, while OpenAI models held #1 in cyber 57% of the time and math 55% of the time.
Robotics benchmarks are still heavily contested.
... that is, except for robotics! While expert-level benchmarks dominates professional work (84%), only 4% of vision, audio and robotics evals can be considered expert-level. 96% of the robotics tests labs headlined this year are everyday tasks most adults can do.
























