Brian Zhou (@brian1zhou) header
Brian Zhou (@brian1zhou) profile picture

maximum plasticity! @harvard & @tjhsst_official.

Joined September 2023
Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)Photo by Brian Zhou (@brian1zhou)
This is exactly the right approach! Big agents are much better at long-horizon physical tasks but aren't fast. Small policies are fast but can't reason and chain actions for long-horizon work. So add them together. The industrial explosion is coming very soon...
We're announcing our first robot-use agent, Waddle-1. It controls bimanual robots on unseen tasks zero-shot. Our agent is the most capable system evaluated by @robocurve, scoring 81 compared to 65 for GPT-6.2 Astra and 59 for Dream-1. It calls LLMs in the cloud to direct lower-level policies that run locally. Waddle-1 is a first step towards agents that combine the capability of reasoning models and the efficiency of reactive policies. We expect such systems to achieve human-speed manipulation in the near future. If you would like to deploy robots with our preview API, please tell us what you want to build here: waddlelabs.ai/api-access
5
527
OpenAI started reporting this benchmark in the GPT 5.6 model cards as an internal benchmark, so it's really cool to see the full results! MedChemBench on Benchmark Almanac: benchmarkalmanac.com/benchma…
MedChemBench is a new benchmark for AI reasoning in medicinal chemistry. It shows the progress our frontier models have made in their ability to help scientists use AI to identify promising drug candidates. openai.com/index/introducing…
1
1
8
521
Brian Zhou retweeted
And as a result, the Tasmanian Government has published one one of the greatest Government Documents I have ever seen
i kinda love how Tasmania has a “Godzilla” like relationship with this guy. “yeah it comes ashore and destroys as much as it can every once in a while but you know, it’s like a spectacle and it leaves eventually”
143
7,600
283
69,417
1,332,871
Gemini 4 Neon now on the Benchmark Almanac: benchmarkalmanac.com/models/…
Introducing Gemini 4 Neon! Pushing the frontier across domains and punching above its model class in a meaningful way. To me, Neon feels like the right balance of intelligence, speed, and usability. Available now across @GoogleAIStudio, the API, and more!
3
312
Don't you ever get sick of Claude or ChatGPT making you the same cookie-cutter websites with Instrument Serif and Inter? I asked Claude for "a font like Starry Night" and it drew one using WhateverFont, a tool that lets you make your own beautiful fonts: fonts.brianzhou.org
4
1
22
958
WhateverFont supports Greek and Cyrillic natively. For Arabic, Hindi, Japanese, Chinese and Korean, WhateverFont traces an open font of that script and draws it again with your font's pen. Here's the snow font in six scripts:
1
1
56
There are some known limitations; right now, WhateverFont does not have handling for true italics (they're handled as slanted letters), hinting, or color fonts that are also variable. Bold weights also need additional rounds of prompting to get right.
1
26
Brian Zhou retweeted
Fable 5.2 didn't build this SF Muni bus backflip game, and it's smaller than the average webpage (1.4 MB) I used Opus 5.5 and SWE-2 Flash instead! Everything you see is only made with code.
Made a lil’ Muni 🚌 ads gallery, send me your designs 🌁 - muni.gallery
17
7
1
49
4,360
Brian Zhou retweeted
AI is getting better faster than our measurements are I threw together SatBench, which measures how quickly major AI benchmarks become saturated (effectively beaten). Of 173 benchmarks cited by the Labs since 2023, 63% are now saturated. Saturation is happening faster each year.
255
74
58
730
77,394
Brian Zhou retweeted
Proud to announce the inclusion of Perceptron's recently released EgoHome fine-grained egocentric video understanding benchmark on Benchmark Almanac! Links below
I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there. Website: benchmarkalmanac.com
2
3
16
646
Brian Zhou retweeted
This is super powerful. The world doesn't really have a good view of all the different models and how they compare to each other, since so many reports cherrypick which evals to run, and indexes like ECI or AAII are selected for various, sometimes opaque, constraints. Great work Brian!
I'm excited to launch Benchmark Almanac! It tracks 3,322 benchmarks from 1,746 leaderboards, 522 launch posts, 346 system cards, and 1,233 open-source GitHub repos. To my knowledge, it's the most comprehensive repo out there. Website: benchmarkalmanac.com
3
4
72
3,214
You can also compare up to four models head to head on every benchmark they've all been evaluated on. Right now, Claude Fable 5.2 beats Gemini 4 carbon, GPT-6 astra and Muse Wonder on 21 of 45 benchmarks all four were evaluated on. benchmarkalmanac.com/compare
1
1
3
286
The almanac is meant to stay current. Every morning, one job reads 26 sources (@latentspacepod ainews, @huggingface daily papers, @arxiv, @epochairesearch, lab model cards, eval hubs, bluesky…) for new benchmarks and models, and opens PRs for maintainers to review. Please reach out if you would be interested in helping maintain this project!
1
3
241
This exists because I wanted an easy way to see: a) what benchmarks and arenas already existed b) how models perform on various aggregate evals c) how benchmarks and evals have changed over time Benchmark Almanac is fully open source. If you see a missing benchmark or score, want to log a known critique (leaked answers, a broken grader), or maintain a benchmark and want it to be featured, please join on as a contributor! Contribute: github.com/brianlzhou/benchm…
3
1
18
1,447
Want to try your hand at making your own blend of benchmarks? Build your own index! You can add/drop evals and give your own custom weighting to each. benchmarkalmanac.com/build
1
1
2
284
Evals also offer a window into a model's jagged capabilities. Anthropic leads 10 of the 13 contested bio and chem benchmarks, while OpenAI models held #1 in cyber 57% of the time and math 55% of the time. Robotics benchmarks are still heavily contested.
1
1
4
341
... that is, except for robotics! While expert-level benchmarks dominates professional work (84%), only 4% of vision, audio and robotics evals can be considered expert-level. 96% of the robotics tests labs headlined this year are everyday tasks most adults can do.
1
1
4
353
Evals have also gotten much harder. Expert and research-frontier tests went from 5% of what labs report in launch posts in 2024 to 38% in 2026 2026 was the year of agents: agent tasks (25% of 2026) passed multiple choice (20%), which was 46% back in 2023
1
1
4
415