Pinned Tweet
The last few months I have been working on a new Benchmark.
Introducing AutomationBench. Trying to measure the cutting edge of model's capabilities in real world business workflows across multiple apps and noisy data.
The best models haven't beat 10% yet.
Opus 5.5 does really well on AutomationBench too, breaking 40%! A major step up for less cost!
Claude Opus 5.5 is here! We ran the numbers.
Here's where you should upgrade from Opus 5 👇️
Any crucial Ops workflows - it scored 62% for Ops at Max effort, at roughly $1.37 per task. Great value for the price.
Overall it scored 40% across 657 tasks - a BIG jump from Opus 5's 26.9%, and one of the best models we've benchmarked yet.
Note: We'd recommend scaling the effort per workflow (instead of keeping the default, Medium, for each). For example, Low scored 48% for Ops compared to Max’s 62%, but Max is double the price. Worth experimenting!
See all the models we've benchmarked here: zapier.com/benchmarks
GPT-6 Sol is here! We benchmarked it on real workflows, so you don't have to.
- Scored 33.2% at xhigh effort - better than GPT-5.6 Sol's 28.77%
- Roughly half the price of 5.6. $2 per million input tokens, down from $4.
- Strongest for: Ops work (scored 58%!)
Upgrade any of your workflows that are using 5.6. It's cheaper AND better (rare combo)
See all the models we've benchmarked here: zapier.com/benchmarks
Replying to @OpenAI
Higher usage limits and lower cost give you more flexibility and room to iterate.
New GPT 6 models! Doing well on AutomationBench.
Replying to @OpenAI
Higher usage limits and lower cost give you more flexibility and room to iterate.
2
62
Daniel Shepard retweeted
AutomationBench (by @Zapier) is the lead benchmark OpenAI used to showcase Astra's workflow automation capability -- 44% at $2.45/task.
Replying to @OpenAI
Astra achieves state-of-the-art results on Agents’ Last Exam, AutomationBench, and ScreenSpot Pro, benchmarks for computer workflow tasks across professions.
GPT 6 Astra scores 41.4% 😳
GPT 6 Astra is here. We ran the numbers on AutomationBench:
It's the highest score we've ever recorded. Clean sweep across every domain.
Scores 41.4% at Max effort. For context, no model had cleared 40% before today (GPT-5.6-Sol scored 28.8%)
𝗕𝗲𝘀𝘁 𝗳𝗶𝘁 𝗳𝗼𝗿: reconciliation, deal review prep, vendor scorecards, anything where touching the wrong record is expensive.
𝗪𝗲𝗮𝗸𝗲𝗿 𝗳𝗼𝗿: outbound comms where the guidance is scattered. Operations and support are its strongest domains. HR is its weakest, same as every model we test (still the new high score, though)
Its edge is arithmetic across messy sources. Finding the policy doc, the logged correction, the exception rule, etc.
Example 1: rebalance a quarterly media budget from last quarter's actuals, with finance adjustments and channel eligibility rules buried in email. Both models produced a budget and landed on the same total. Astra found the adjustments, so every per-channel number was right. Sol's looked finished and had the splits wrong.
Example 2: answer and log 15 integration inquiries using a reply standard stored in a doc. Astra searched, could not find the standard, and stopped. Zero replies sent. Sol did not find it either, took its best shot at all 15, and earned partial credit.
Those examples highlight how these two models make tradeoffs... Astra will not guess. When the instructions exist and it can find them, it finishes the whole job. When it cannot, it pauses the work instead of improvising.
Crazy week for LLM releases after a few quiet ones. Astra isn't available to the public yet, but should be soon.
We run every new model through @Zapier's AutomationBench, 657 of the hardest workflows we have, across finance, HR, marketing, operations, sales, and support.
See every model and every score here: zapier.com/benchmarks
4
79
Fable 5.1 is out today. It takes the lead and scores 31.4% on AutomationBench.
Claude Fable 5.1 is out today.
𝗪𝗵𝗮𝘁 𝗶𝘁’𝘀 𝗯𝗲𝘀𝘁 𝗳𝗼𝗿:
Policy-heavy approval workflows.
Ex: “route pending expense reports, raise the auto-approve ceiling in the approval matrix, correct per diem amounts to company policy”. Fable 5 scored 0%, Fable 5.1 scored 100%.
Same story across 13 finance tasks (policy conflicts, approval hierarchies, decoy data, etc). It's way better at digging up the actual policy and not getting fooled.
@Claudeai's Fable 5.1 also improves on Support (18% -> 29%), Marketing (17% -> 27%), and Sales (11% -> 21%)
𝗔𝘃𝗼𝗶𝗱 𝗳𝗼𝗿: Routine cross-app coordination. That seems to have taken a backseat to higher level reasoning work with this update. For Operations work, Opus still wins (50% compared to Fable 5.1’s 36%)
(Also avoid for pipeline analysis and market research: scored a 0 on both)
Quick caveat: due to Fable’s safety classifier, some steps are refused. When that happens, Opus steps in to complete the task. That’s what “Opus fallback” means in the graphic below.
A model upgrade used to mean everything got a little better at once. Now, as they become more specialized, it’s crucial to know where to use each.
Try Fable 5.1 in @Zapier workflows with hard judgment calls that need a lot of reasoning
Daniel Shepard retweeted
Love seeing @Zapier's AutomationBench as the first benchmark listed.
Congrats on taking top spot, @GeminiApp and @OfficialLoganK!
Gemini 3.7 Flash takes the lead with 30.4%on AutomationBench!
Usually you have to pick: best model or cheapest model
Gemini 3.7 Flash is both, and it’s available in Zapier!
• 30.4% on AutomationBench (new #1)
• Less than half the price of GPT 5.6 Sol and Opus 5
• 11.7 points ahead of every other model on finance workflows
More below:
Daniel Shepard retweeted
one thing that has saved my projects more time than I can count is evals
boy was I excited when florian, quite literally an expert in benchmarks, agreed to hop into a ~2h interview to do a walkthrough of what the eval landscape looks like in 2026
(and also answer my personal business questions on the subject)
given that now running frontier model through benchmarks is a vector for hacking other systems in order to avoid doing work (looking at you sol), I think it's more important than ever to educate folks on the evals situation.
had a lot of fun throughout this session and I hope that you learn a thing or two!
enjoy! 🌹
table of content:
0:00:00: are AI Benchmark broken?
0:05:45: Florian Brand background
0:09:00: what motivates florian to work on evaluation?
0:13:33: what is the mirrorcode benchmark about?
0:18:20: cheating in agent benchmark is insaneeeee
0:24:08: LLM benchmarks in era of agents
0:26:30: what’s up with the pelican man
0:28:27: evals are about capabilities
0:31:46: components of running evals
0:35:30: the volume of things to audit is huge!!!
0:40:20: expert answers are wrong hahahaha
0:46:00: api providers aren’t the same
0:48:00: benchmark narrow capabilities (synthetically)
0:50:56: link between eval and environment
0:53:45: small validated benchmark or massive bench?
0:56:11: what is your flow to review a benchmark?
0:58:30: tracking work capabilities with evaluation
1:00:20: slide deck in industry is all vibecoded
1:03:30: harness impact in the evaluation
1:07:39: hardware/sandboxes impact evaluation too!
1:11:00: “is it going to get worse?”
1:12:40: all components influence the final score
1:13:50: training models on different harnesses?
1:17:20: is the model just the weights or it’s all of it?
1:19:30: how to craft benchmark that prevent to cheating and undereliciting models in 2026
1:23:19: ways agents cheat and steal
1:26:00: correct elicitation of capabilities is important
1:36:00: building evaluation on prime intellect
1:45:10: how do you design interactivity benchmarks?
1:48:40: do you think evals are well set to reflect real world performance?
1:52:50: what will the benchmarking landscape will look like in 1 year
Opus 5 just took AutomationBench Leaderboard with all 5 of its reasoning levels. 26% 😲Even on Low Reasoning it is higher than all the competition.
Gemini 3.6 Flash takes the leaderboard above GPT 5.6 Sol and Fable 5 (strict classifier)! Gemini 3.6 Flash is surprisingly good at knowing when NOT to act.
It is more efficient and calls tools in parallel like other models vs 3.5 Flash not.
Nice work @OfficialLoganK and team!
Kimi K3 tops AutomationBench on Artificial Analysis! 😯
artificialanalysis.ai/evalua…
GPT 5.6 is fully out now and Sol (18.1%) is pretty comparable to Fable (17.4%) on AutomationBench!
Cool that Grok 4.5 is #1 in some respects, even with respect to Fable 5
SpaceXAI's Grok 4.5 takes the #1 spot on AutomationBench-AA with a score of 51%, ahead of Claude Fable 5 (49%) and Claude Opus 4.8 (48%) at roughly a quarter of their cost per task - the first model to complete more than half of workflow objectives without breaking any business rules
AutomationBench-AA, our independent leaderboard for @zapier’s AutomationBench, tests whether AI agents can automate real SaaS workflows while adhering to business rules. The test set is private to prevent contamination.
Models complete 657 tasks across 40 simulated app environments including Gmail, Google Sheets, Slack, Salesforce, and HubSpot, and the headline score is the share of objectives completed without violating any guardrails.
Key takeaways:
➤ Grok 4.5 completes more objectives than any other model: It completes 79.9% of task objectives and strictly passes 21.9% of tasks. This is the highest we’ve measured on both outcomes, exceeding Claude Fable 5’s 73.3% objective completion and Claude Opus 4.8’s 19.3% of fully-completed tasks
➤ Grok 4.5 pushes out the Pareto frontier of score vs. cost per task: At $0.34 per task, it is both cheaper and higher-scoring than every other leading model - Claude Fable 5 ($1.35 per task), Claude Opus 4.8 ($1.46), GPT-5.5 (xhigh, $1.28), and Gemini 3.5 Flash (high, $0.49)
➤ It is extremely token-efficient: Grok 4.5 uses ~8k output tokens per task, the fewest of any leading model - less than a quarter of Claude Opus 4.8 (32k) and a third of Gemini 3.5 Flash (24k). Its total token usage of 0.44M per task is among the lowest on the leaderboard. Low cost is driven by this efficiency as well as low token pricing
➤ Grok 4.5 uses fewer turns with many parallel tool use: Grok 4.5 resolves tasks in ~16 turns, fewer than GPT-5.5 (xhigh, 25) and less than half of Gemini 3.5 Flash (high, 35), while making the most tool calls per task of any leading model (52.5). It batches 3.3 tool calls per turn, compared to ~2.5 for Claude Opus 4.8 and ~2.0 for GPT-5.5 (xhigh)
➤ Guardrails still get broken: Grok 4.5 triggers 0.63 violations per task, above Claude Opus 4.8 (0.55) and Gemini 3.5 Flash (0.46). At 13.0 objectives completed per violation, it trails Gemini 3.5 Flash (15.0) and Claude Opus 4.8 (13.5)
➤ Its strongest lead is in the hardest domain: Grok 4.5 completes 71% of Finance objectives, the domain with the lowest average score, ahead of Claude Fable 5 (64%) and Claude Opus 4.8 (62%)
Congratulations to @spacexai and @elonmusk on topping the leaderboard!




































