
working with incredible humans at @officehours. partner in crime to @zokopp
San Francisco, CA
Joined August 2013
- Tweets809
- Following609
- Followers404
- Likes2K
joe kim🗽 retweeted
With Astra 6.2 on the scene, here's a deeper look at what's driving the overall score for each of the top models.
Carbon edges out competition and is very well rounded. The model excels in biochem and biophysics applications, is strong in neuroscience and genetics, and puts the only passing task on the board in immunology. Trails most in cancer biology and and physiology.
joe kim🗽 retweeted
The newest frontier models are advancing but all still struggle with interpretation in biology when given complex challenging science questions that have input data that biologists use and ask for multiple things just like experts ask while running experiments, doing analysis, troubleshooting, trying to extrapolate what everything means 🙂. We are working on many such projects with our scientists and happy to send questions, let us know :) scientific taste is earned through years of expert research (and “failure” aka things not working in lab) 🙏. More exciting work coming soon…
horizon.officehours.com/rese…
GPT Astra 6.2 is a very capable model for long-horizon Life Sciences and Biology work.
Incredible work and congrats to @joyjiao12, @yunyunash and the @OpenAI team.
OpenAI enters the top 5 on BioResearchBench.
horizon.officehours.com/rese…
joe kim🗽 retweeted
GPT Astra 6.2 is a very capable model for long-horizon Life Sciences and Biology work.
Incredible work and congrats to @joyjiao12, @yunyunash and the @OpenAI team.
OpenAI enters the top 5 on BioResearchBench.
horizon.officehours.com/rese…
joe kim🗽 retweeted
My overall read is that this shows a monster performance by hydrogen to handle real-world complex biology work. Scientists should use hydrogen for most things and use Carbon for the heavy lifting, long-horizon work (cheaper and significantly better than Fable 5.2)
joe kim🗽 retweeted
Interestingly, hydrogen on average misses the fewest rubrics items (~16%) vs. all other models except for Carbon (~15%) missing fewer as a % of total criteria than Fable 5.2 (~17%) and Opus 5.5 (~18%)
joe kim🗽 retweeted
We’re continuously adding new models to the BioResearchBench leaderboard!
Results for more cost effective models, including Gemini 4 Hydrogen and Muse Ember, are now available!
In the last week, @GoogleDeepMind and @Meta each released workhorse models that have set the cost-performance frontier for long-horizon biology work.
Hydrogen's performance is especially surprising and great to see! It was very close to Opus 5.5 on most tasks but (unsurprisingly) had some challenges handling the very long-horizon tasks leading to time-bound failures.
Read more: horizon.officehours.com/rese…
joe kim🗽 retweeted
This is why we built an eval that has expert-verified flexible grading. In order to test and capture real scientific reasoning capabilities you have to go beyond testing just tool use and reasoning over data.
In BioResearchBench we test tool use and computational capabilities but we also attempt to capture more true scientific reasoning like knowing when a signal is an assay artifact rather than biology, whether the controls rule out other explanations, when a “candidate” can honestly be called “validated,” and which experiment to run next.
horizon.officehours.com/rese…
joe kim🗽 retweeted
In the last week, @GoogleDeepMind and @Meta each released workhorse models that have set the cost-performance frontier for long-horizon biology work.
Hydrogen's performance is especially surprising and great to see! It was very close to Opus 5.5 on most tasks but (unsurprisingly) had some challenges handling the very long-horizon tasks leading to time-bound failures.
Read more: horizon.officehours.com/rese…
joe kim🗽 retweeted
🚀🚀🚀
Google takes the lead!
Just 2 days after the release of Fable 5.2, Carbon is the new SOTA on BioResearchBench.
Congrats @divy93t and team.
joe kim🗽 retweeted
Pushing the frontier, one element at a time!
Meet Gemini 4 Hydrogen: built for rapid coding, multimodal synthesis, and everyday workflows.
It finishes tasks 2.5x faster than 3.8 Flash and surpasses Argon on the @ArtificialAnlys Intelligence Index.
joe kim🗽 retweeted
Something to be proud of!
Google takes the lead!
Just 2 days after the release of Fable 5.2, Carbon is the new SOTA on BioResearchBench.
Congrats @divy93t and team.
joe kim🗽 retweeted
Replying to @GeminiApp
@GeminiApp and @AnthropicAI—nice job on our BioResearchBench Eval where questions were written from phenomenal scientist experts in diverse bio subdomains 😊. Reasoning in biology is challenging for models, understandably so! Gemini’s newest models like Carbon have been really impressive in biology, highly recommend that scientists give it a try if you haven’t in your research 🚀 (also really great in troubleshooting experiments that aren’t working, snap a pic and ask Carbon).
Google takes the lead!
Just 2 days after the release of Fable 5.2, Carbon is the new SOTA on BioResearchBench.
Congrats @divy93t and team.
joe kim🗽 retweeted
Google takes the lead!
Just 2 days after the release of Fable 5.2, Carbon is the new SOTA on BioResearchBench.
Congrats @divy93t and team.
Fable 5.2 is the new SOTA model on BioResearchBench, showing a 5 p.p. improvement over Opus 5.5., including solving 5 tasks previously not solved by any model. Congrats to
@AnthropicAI @amirshanehsaz @mgdurrant @nc_frey @JCoolScience and team
joe kim🗽 retweeted
Fable 5.2 ranks #1 on Bio Research Bench 🚀
Fable 5.2 is the new SOTA model on BioResearchBench, showing a 5 p.p. improvement over Opus 5.5., including solving 5 tasks previously not solved by any model. Congrats to
@AnthropicAI @amirshanehsaz @mgdurrant @nc_frey @JCoolScience and team
joe kim🗽 retweeted
Fable 5.2 is the new SOTA model on BioResearchBench, showing a 5 p.p. improvement over Opus 5.5., including solving 5 tasks previously not solved by any model. Congrats to
@AnthropicAI @amirshanehsaz @mgdurrant @nc_frey @JCoolScience and team
Introducing BioResearchBench, a new benchmark focused on scientific reasoning in biology from @OfficeHours built by 100+ PhD biologists in our network.
For models and agents to materially speed up scientific discovery, scientists need to be able hand off work that would take them hours to complete, come back, and find it done right without mistakes.
This work shows that frontier models aren’t there yet. They still struggle to stitch multiple steps together, get the computations right, interpret the data correctly, handle uncertainty, and draw the right conclusions.
We built 50 long-horizon research tasks across 10 subdomains, from genomics and cancer biology to neuroscience and immunology. Each uses multiple data files, takes an expert hours to several days to complete, and is independently reviewed by at least 2 PhD biologists for verifiability and scientific accuracy.
Every frontier model passes fewer than 1 in 5 tasks:
Claude Opus 5.5: 18%
Gemini 4 Argon: 13%
Muse Wonder: 11%
GPT-6 Astra*: 4%
Grok 4.7: 4%
The top models met ~80% of individual rubric criteria, but completed the full end-to-end workflow correctly <20% of the time. Half of the tasks were not solved by any model.
The biggest gaps point to where models need to improve next: computation and data handling (failures on 67% of tasks), handling uncertainty (55%), and reporting complete answers (55%).
An important note on scoring: GPT-6 Astra’s score is skewed lower because every task had to stump Astra (SOTA at the time) during development before it made the final set.
Paper, leaderboard, and per-task results:
horizon.officehours.com/rese…
@samar_abedrabbo @alexpryhoda
How well can frontier models perform real-world biological research?
We tested 5 leading models on 50 challenging research tasks developed by 100+ of our PhD biologists. The best model passed just 18%
check out full results here:
horizon.officehours.com/rese…
Introducing BioResearchBench, a new benchmark focused on scientific reasoning in biology from @OfficeHours built by 100+ PhD biologists in our network.
For models and agents to materially speed up scientific discovery, scientists need to be able hand off work that would take them hours to complete, come back, and find it done right without mistakes.
This work shows that frontier models aren’t there yet. They still struggle to stitch multiple steps together, get the computations right, interpret the data correctly, handle uncertainty, and draw the right conclusions.
We built 50 long-horizon research tasks across 10 subdomains, from genomics and cancer biology to neuroscience and immunology. Each uses multiple data files, takes an expert hours to several days to complete, and is independently reviewed by at least 2 PhD biologists for verifiability and scientific accuracy.
Every frontier model passes fewer than 1 in 5 tasks:
Claude Opus 5.5: 18%
Gemini 4 Argon: 13%
Muse Wonder: 11%
GPT-6 Astra*: 4%
Grok 4.7: 4%
The top models met ~80% of individual rubric criteria, but completed the full end-to-end workflow correctly <20% of the time. Half of the tasks were not solved by any model.
The biggest gaps point to where models need to improve next: computation and data handling (failures on 67% of tasks), handling uncertainty (55%), and reporting complete answers (55%).
An important note on scoring: GPT-6 Astra’s score is skewed lower because every task had to stump Astra (SOTA at the time) during development before it made the final set.
Paper, leaderboard, and per-task results:
horizon.officehours.com/rese…
@samar_abedrabbo @alexpryhoda



















