@Stanford with @HazyResearch and @Scott_linderman. Working on data and evals for ๐ค๐ง PhD @cu_neurotheory @GoogleDeepmind
San Francisco, CA
Joined July 2011
- Tweets1.3K
- Following2.6K
- Followers1.9K
- Likes37.9K
Pinned Tweet
Very excited to release Terminal-Bench 2.1!
Coding agents are among the most economically consequential deployments of LLMs to date. As agents improve, benchmark reliability matters more.
We audited TB2.0 and found and corrected issues in 28/89 tasks. 30% of the benchmark!
But the rankings survived, absolute scores moved up to 12pp!
Kelly Buchanan retweeted
GPT-Rosalind surfaced a new lead for drug discovery: could restoring cholesterol transport protect neurons from degeneration?
DCTN1 is a gene linked to Perry syndrome, a rare neurodegenerative disorder. In mouse-brain data, disrupting it increased expression of genes involved in making cholesterol. GPT-Rosalind suggested a possible explanation: cholesterol may not be reaching where neurons need it.
That gives scientists a specific mechanism to test and, if confirmed, a direction for developing new treatments for neurodegenerative disorders.
With @perturbai_tx and @scrippsresearch, a team of thousands of GPT-Rosalind agents worked in parallel, connecting gene-expression results from millions of mouse brain cells with scientific literature in just 32 minutes. Weโre now expanding this collaboration to identify new drug targets.
openai.com/index/scaling-ai-โฆ
Kelly Buchanan retweeted
Hello again, it's been a while.
We're writing to let you know that we have raised quite a lot more money. We're letting you know because we are on the lookout to hire smart people and this will help us do that.
See you again when we have a product or have raised more money!
Weโve spent the past year developing new primitives for GPUs. We believe this enables new algorithms to change how machines learn, open new markets, and dramatically reshape AIโs cost structure.
We have raised a $1 billion Series A, co-led by Index and Lux.
Kelly Buchanan retweeted
Weโve spent the past year developing new primitives for GPUs. We believe this enables new algorithms to change how machines learn, open new markets, and dramatically reshape AIโs cost structure.
We have raised a $1 billion Series A, co-led by Index and Lux.
Kelly Buchanan retweeted
We formally announced Mirendil less than 4 months ago. Today, I'm excited to share an important milestone for:
Pretraining at scale without a pretraining team.
Mirendil's mission is to solve Recursive Self-Improvement (RSI) and use it to democratize frontier AI development, ultimately accelerating science and technology. At Mirendil, when we think about RSI, we think holistically about the company as an intelligent entity that self-improves toward its goal. From that perspective, the most powerful notion of acceleration is reducing the time and resources (number of human experts, compute, and other costs) needed to achieve ambitious goals.
Given that pretraining is perhaps the most important yet most challenging part of frontier AI development, we decided to run an ambitious controlled experiment: Starting fully from scratch, can we significantly reduce the time, number of human experts, and compute resources needed to go all the way to successfully pretraining a strong coding model at scale?
Mirendil-Base is the result of this experiment. Our first pretrained model was built by one researcher in 3.5 months, with less than $10M in compute. This includes all the research and small-scale experiments it took to get there, as well as the final training run.
The recipe holds up as the models get bigger. For reference, Mirendil-Base is in the 100B-500B parameter range, Mirendil-Base-Large in the 500B-2T range, and Mirendil-Base-XL in the 2T-10T range.
We're beginning to see a world where a single researcher, empowered by AI, can take on work that once required an entire research organization. As these systems improve, the bottleneck shifts from the size of a team to the ambition and judgment of the people directing them. This has profound implications for who gets to build AI and, ultimately, who gets to push the frontiers of science and technology forward.
Weโll share independent evaluations, post-training benchmarks and API-access sooner than you think so stay tuned!
Kelly Buchanan retweeted
New on our Science Blog: designing protein binders against scorpion neurotoxins.
Scorpions sting about 1.2m people every year, and antivenoms are often costly or in short supply. Claude has made an early step toward finding new treatments.
anthropic.com/research/claudโฆ
Kelly Buchanan retweeted
At @Arda_Industries, weโre putting AI to work in the physical world to speed up American manufacturing. Proud to work alongside the incredible people at Arda who share this mission.
Build more things! ๐บ๐ธ๐
We started Arda in 2025 to speed up American manufacturing. We have spent the last year building with Anduril as our design partner. Today we are launching the company publicly on CNBC and announcing our Series A: $160M at a $1.5B valuation.
Our platform helps manufacturers design and operate manual assembly lines, putting designs into production faster now. Next we'll build automated lines that can be reconfigured as quickly as human workers, letting America make products that now require armies of workers in Asia.
Kelly Buchanan retweeted
Introducing Omnii-1, our most powerful genome language model, and a major step toward general biological intelligence
In a world first, Omnii-1 designed the first AI-generated bacteriophages to be administered to a human patient with an antibiotic-resistant infection under compassionate use
Kelly Buchanan retweeted
We looked at TerminalBench 4.0 and found a lot of issues. After fixing the issues Sonnet 5.5 scores 89.2%. So, TerminalBench is already mostly saturated.
Along the way we found a lot of crazy stuff, including reward hacks in the published transcripts.
Kelly Buchanan retweeted
We trained our Muse foundational models, including Muse Wonder, at Prometheus, our gigawatt-scale supercluster in New Albany, Ohio.
Across a supercluster with hundreds of thousands of GPUs, even one underperforming host can slow an entire training run. We engineered Prometheus for resilience at scale, achieving 85%+ goodput on pretraining, 90%+ on reinforcement learning and ~47K GPU hours mean time between interruptions during Muse Wonder training.
Take a look at the tech powering Prometheus: go.meta.me/4yXXApZ
Kelly Buchanan retweeted
๐ DeepSeek-V4.1-Flash-1017 is here!
Stronger benchmark performance, faster task completion, and shorter outputs than the previous versionโa Pareto improvement in performance and cost. Now live on our website, app, and API.
๐ ๏ธ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model.
1/4
Kelly Buchanan retweeted
Something I find strange about parts of the ML community is how we entertain the idea that the systems we're building might be fundamentally beyond our understanding.
This is very different from the perspective I'm used to in physics and especially in biology (1/)
Kelly Buchanan retweeted
Today weโre releasing Muse Spark Open, the most capable US open-weight model to date.
Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning.
Key Model Specs:
โ 816B total / 42B active parameters
โ 1M token context window
โ Natively multimodal across text, image, video, audio
Kelly Buchanan retweeted
Weโre exploring how AI can accelerate health science research.
While AI has made significant breakthroughs in formal sciences like mathematics, health data science depends on imperfect real-world data and consequential choices about cohorts, variables, and causal assumptions.
To explore how AI can assist health researchers, we built a human-led research scaffold powered by Muse Spark 1.3 and Muse Wonder with access to 5M retrospective de-identified electronic health records and publicly available data. A cohort of medical experts used the system to conduct original research, accelerate hypothesis generation, code execution, and manuscript drafting while maintaining full authority over scientific decisions.
Today, weโre sharing 10 research manuscripts produced through this exploration, spanning cardiology, oncology, infectious disease, emergency medicine, and more.
Read more in our research blog: go.meta.me/3U06wvD
Kelly Buchanan retweeted
How can the math community forge the best path forward in the AI age? We're committing $15M to workshops and initiatives โ plus free AI computing resources for foundation-supported researchers โ through new partnerships with @AnthropicAI and @OpenAI: bit.ly/4zpfIbW
#math #science #AI
Kelly Buchanan retweeted
Mercor Research mapped AI benchmarks to actual US jobs.
Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark.
Today, weโre releasing the ๐๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ ๐ ๐ฎ๐ฝ ๐ผ๐ณ ๐๐ต๐ฒ ๐จ๐ฆ ๐๐ฐ๐ผ๐ป๐ผ๐บ๐, an interactive breakdown of which economic tasks are covered by AI benchmarks.
Benchmarks are how we measure AIโs progress and anticipate its impact on real work. Theyโre crucial to understand which jobs AI can automate, where the biggest gaps are, and how those capabilities are changing over time.
The Benchmark Map is our first step toward building a resource that makes it easier to answer: ๐๐ฐ๐ฆ๐ด ๐ต๐ฉ๐ช๐ด ๐๐ ๐ฎ๐ฐ๐ฅ๐ฆ๐ญ ๐ธ๐ฐ๐ณ๐ฌ ๐ง๐ฐ๐ณ ๐ฎ๐บ ๐ฐ๐ค๐ค๐ถ๐ฑ๐ข๐ต๐ช๐ฐ๐ฏ?
Explore it here: mercor.com/research/labor-inโฆ
Check of this work led by @KumbongHermann and @ishanskhare !
With Next-Concept Modeling, we watch language models build up text, from a rough sketch to the finer details
๐ Introducing Next-Concept Modeling (NCM): our new paradigm for fast, parallel, coarse-to-fine text generation.
Compared to strong diffusion language models (DLMs) at similar parameter size, NCM achieves:
โก Up to 31ร faster generation on downstream tasks like summarization and machine translation with comparable or better quality.
๐๏ธ 10โ21ร faster unconditional generation across TinyStories, LM1B, and OpenWebText, with comparable or better generative perplexity and using 3โ5ร fewer training tokens.
๐งฉ Better performance on reasoning tasks like Sudoku, outperforming both AR and diffusion baselines with fewer sampling steps.
With @ishanskhare, @IddahMlauzi, @ekellbuch, @simran_s_arora, @Azaliamirh, and @HazyResearch.
More details in the thread ๐งต๐
Kelly Buchanan retweeted
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Kelly Buchanan retweeted
Introducing ThunderSync, a new approach to RL and OPD on long-horizon agentic tasks.
Post-training methods like GRPO and OPD on long-horizon tasks leave trainer engines idle during rollout, verification, and reward measurement.
To avoid this, asynchronous approaches overlap rollout and training across updates, but policy staleness reduces accuracy.
ThunderSync is a new method to squeeze out these bubbles in agentic post-training without policy staleness. For GRPO, we compute each trajectoryโs score gradient as soon as its reward is ready, without waiting for the group. For OPD, we compute gradients for each turn as soon as the teacher scores it, without waiting for the full trajectory, so training overlaps with tool execution and later turns. We show that this is exactly the same update as batch-synchronous training.
The result is up to 1.9x faster RL while matching the performance of synchronous updates.
Paper: arxiv.org/abs/2610.05935
Code: github.com/seilk/thundersync
Blog: seilk.github.io/thundersync
Introducing ThunderSyncRL โก It makes agentic RL up to ๐ญ.๐ตร ๐ณ๐ฎ๐๐๐ฒ๐ฟ with ๐๐ฒ๐ฟ๐ผ ๐ฝ๐ผ๐น๐ถ๐ฐ๐ ๐๐๐ฎ๐น๐ฒ๐ป๐ฒ๐๐.
We observed that synchronous GRPO/OPD leaves learner GPUs idle for long stretches (๐ฐ๐ต.๐ฑ% of step time when training on SWE-smith). Asynchronous training avoids this, but it learns from ๐๐๐ฎ๐น๐ฒ ๐ฝ๐ผ๐น๐ถ๐ฐ๐ถ๐ฒ๐ and performs worse. Our goal was to keep GPUs maximally utilized while still doing strictly on-policy updates.
ThunderSyncRL streams gradients as soon as their inputs are ready, then takes one synchronous optimizer step. We prove the GRPO and OPD updates are identical to batch-synchronous training.
On SWE-bench Verified and Terminal Bench 4.0, it reaches Sync's pass@3 up to ๐ญ.๐ตร ๐ณ๐ฎ๐๐๐ฒ๐ฟ and beats Async by up to ๐ฎ.๐ฐ๐ณ ๐ฝ๐ผ๐ถ๐ป๐๐ of pass@3 at the same GPU budget.
Grateful to my co-lead @hangoo_kang and to the amazing @Azaliamirh, who made this work possible.
And a big thank you to @TarunSures41845, Youngeun Kim, @sayshrey, and Seong Jae Hwang for building it with us. (1/7) ๐งต
๐ Paper: arxiv.org/abs/2610.05935
๐พ Code: github.com/seilk/thundersync
๐ Blog: seilk.github.io/thundersync