And here it is, our Fable 5.2 benchmark, and a new feature, a thinking report! 🧠
Can confirm, no better way to start a Saturday than a fresh benchmark 🖖
Okay, the day has come, my Fable 5.2 Benchmark with @VulcanBench is now complete 🖖
This is the first model to clear every task, and thus, am calling it, "the Frontier v4 killer"
And honestly, if something was going to kill this eval suite, it's an honor to have Fable be the one to do it.
That being said, VulcanBench is all about testing models on real engineering tasks, that teams like mine would use a model for every day, to really understand first, if you need the model at all, and second, what effort level you need if you do.
I say anything in the 80 - 90 range on Frontier v4 is strong enough for daily work, and that is why, if you use Fable 5.2, Low effort is likely all you need for routine daily engineering tasks.
But you'll pay a price, as you can see from the cost-per-task chart, it isn't cheap. But it does think in amazing ways.
Which is why I'm also introducing a brand new section on VulcanBench where I share analysis on how a model thinks...and yeah, hard to think of a better model to kick this off with than Fable.
With that, here's how you access the benchmark and the thinking report if you want to do a deeper dive into either or both.
VulcanBench on Frontier v4:
vulcanbench.com/benchmarks/s…
How Fable 5.2 Thinks Report:
vulcanbench.com/reports/01-h…
Live long and benchmark 🖖
2
9
650
Might delay this by a day or two, still reviewing traces, and putting together a new deliverable from VulcanBench that I'm probably going to call:
Thinking Reports 🧠 📄
Looking at the traces for a model like Fable is something that for me, has become in many ways like appreciating art.
That might sound overly dramatic, but it really does feel that way.
And while I know, of course the most interesting thing to most ppl is the actual benchmark report, i.e. showing how a model performs from an accuracy, cost, and timing standpoint...maybe there are other people like me that want to understand more about how the model thinks when it's solving a task in an eval suite, and how this varies from model-to-model.
I can think of no better model to start this "Thinking Reports" project with, than Fable 5.2.
So expect to see this sometime over the weekend, going to take more time to really do this right.
Live long and study how models think 🖖
Fable 5.2 benchmark results are in, reviewing the traces in detail now.
Publishing model cards in the morning.
You thought Ocarina of Time was a masterpiece, wait until you see the traces for this one 🖖
whoa, muse spark 😳
Frontier models in 2026 cheat on tasks, even when the requirements are unambiguous, and the cheat is obviously antisocial.
In HoneyBench v0.2, we've observed models:
• Trying to break out of our evaluation containers.
• Seriously injuring simulated cancer patients to hit treatment goals.
• Compromising a power grid’s safety to meet a reliability target.
• Concealing their use of available cheats and presenting their solution as honest.
All of the models we measured cheated, from 12% of runs (Fable 5.2) to 86% (Muse Spark Open).
1
5
680
Early results with Fable 5.2, and wow, just wow, this is one heck of a model.
After seeing how it does at Low effort, I think it's safe to say, this is the model that will retire our Frontier v4 eval suite, it's been a good ride, now to finish Frontier v5 🖖
Just got access to Fable 5.2 today, and immediately started benchmarking on @VulcanBench w/Frontier v4, safe to say, I think Fable 5.2 will mark the official sunsetting of this eval suite.
I've been working on Frontier v5 for over a month, getting closer, but not there yet.
That being said, just look at Fable 5.2 on Low effort compared to Fable 5.1. From ~26 min/task down to ~5 min/task, and from $183 to $41, this is a truly amazing model. Can't wait to do a deeper dive into the traces to see how it thinks under-the-hood.
More to come, this is just the first effort level, but I think Fable 5.2 will very likely get a perfect score on every one, which means, it's the model that puts Frontier v4 into retirement.
It was a good eval suite, but it has met it's match.
Live long and benchmark 🖖
2
26
879
One of the coolest new benchmarks I've seen.
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
Which harness should you run your open model in? I ran 28,849 trials to find out.
The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
@opencode costs $3.35 per solved task, 11.7 times as much as Pi.
See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
Leaderboard: huggingface.co/spaces/joelni…
Harness Lens: huggingface.co/spaces/harnes…
4
640
Credits refreshed, Muse Wonder Full benchmark completed, details below:
Okay, no better way to start a Friday than with some fresh benchmark results right?
When I first ran my Muse Wonder Full benchmark with @VulcanBench I didn't have enough credits to run Max.
But my subscription renewed, and I was able to finish the benchmark and run Max, so now have the complete benchmark finished, 115 runs, 23 tasks each.
Two nuggets 🔽
1. Increasing effort level doesn't really make much of a difference in accuract: Muse Wonder Full scores between 87.0 and 88.4 at every level. Going from Low to Max adds about 1.4 points but takes 50% longer (27 to 41 minutes) and costs 50% more (about $3.93 to $5.91 a task).
2. GPT-6.1 Sol pretty much ties it for a lot less. At Max, Sol scores 88.3 vs Muse Wonder Full’s 88.4, at about a thirteenth of the cost ($0.46 vs about $5.91 a task) and in about a third of the time (15 vs 41 minutes).
My final assessment now that the data is in: the Meta DS team has made awesome progress with Muse, massive improvements, but still not a model I would use for routine daily coding tasks when I can get the same accuracy, faster and cheaper with a model like GPT-6.1 Sol.
All data live on the site now:
vulcanbench.com/benchmarks/s…
And as always, all the traces, available on Github:
github.com/morganlinton/vulc…
Live long and benchmark 🖖
5
383
I have been looking at expanding VulcanBench beyond agentic coding, and now have a clear direction I want to move in: Computer Use.
Personally, I think Computer Use is going to be one of the most useful things we do with agents over the next 3-5 years, and I also think it's something we'll want to do as fast and token-efficiently as possible, without sacrificing accuracy.
Here's a bit more about what I'm thinking, a "Morgan decided to build an eval suite for Computer Use" 🧵
🤖 Made with AI
Adding two new numbers that I track:
1. screenshot tax (pass rate with pixels only vs. pixels plus an accessibility tree)
2. a safety track (injection compliance and unsafe-action rates, reported next to task success).
1
80
None of this is built yet, still in the planning phases, and safe to say I have a lot to learn as I go here.
If you run computer-use agents, I would love to hear more about what bugs you the most and where you think I might need to go deeper with this eval.
And if there's an existing benchmark you think does a really good job already with Computer Use, please let me know, I'd love to learn from them too!
Live long and make more benchmarks 🖖
1
2
77
Woke up thinking about model routing, and the direction many companies are going.
Today most individuals and small(ish) teams use subscriptions with pretty heavily subsidized token costs.
Larger companies moved to Enterprise pricing which is often 5x - 10x what individuals or small teams pay.
This has led to the creation of model routers, essentially solutions that optimize the models engineering teams use and save 30% - 50% off Enterprise pricing.
Often, the key unlock here is for large teams, a lot of engineers just use one model for planning, coding, and code review, often at High effort, because they don't want to think about things like model and effort level.
Model routers often use different models, and effort levels at each of these stages, and (maybe) change what they use based on task size/difficulty/language, but I'm not so sure about that.
Starting to noodle on evaluating model routers to learn two key things:
1. Are model routers actually changing model and effort level based on the complexity of the task, size of the repo, programming language, etc. or do they just use the same models/effort levels for everything?
2. Sure, you're saving off of Enterprise pricing if you have hundreds of engineers just using the same model at High effort (or higher) for everything, but how much more are you paying vs using a subscription?
Had a good convo with Fable about this, put together a plan, starting to build. More to come 🖖
After initial testing, we have decided not to benchmark Kimi K3.1 Pro, more details below:
Okay, so I have decided not to benchmark Kimi K3.1 Pro with @VulcanBench after running some initial tests.
This is one of the interesting and complex things about running these benchmarks. At the end of the day, as a CTO, I care about models that me and my team would use for daily coding tasks.
And this means models need to have a good balance of accuracy and cost efficiency.
I started testing Kimi K3.1 Pro on low effort, with four tasks, and it solved none of them, took 26 minutes per task, and cost $2.05.
For comparison, GPT-6.1 Sol low, on the same four tasks solved three out of four, took 4.5 minutes per task, and cost $0.21.
This is also a really good example of why you can't just look at input/output token cost to judge the cost of a model.
Kimi K3.1Pro and GPT-6.1 Sol both cost $2/M input tokens and $10/M output tokens.
What input/output cost doesn't take into account is how many tokens a model uses per task, and in this case, Kimi K3.1 Pro uses ~10x the tokens, so it's actually ~10x more expensive than GPT-6.1 Sol.
So I'm going to pass on benchmarking this model, it would cost me between $200 - $300 to do it, and I already know this isn't a model I'd use with my team.
Live long and benchmark, thoughtfully 🖖
1
8
611
Some Sunday morning thoughts 🖖
This morning I'm working on a new eval suite for @VulcanBench, Routine v2.
I can't stop thinking about a gap that exists in benchmarking today.
And I don't think I'm effectively filling that gap yet.
I think what me, and many other benchmarks find ourselves doing is trying to challenge the frontier.
To be clear, I think we should all keep doing this. I know for me and many others, building eval suites that truly challenge the ever-changing frontier is one of the most interesting things you can do right now.
It is very likely, the hardest thing I've ever done in my life, and I'm really loving how much harder it gets each month. In many ways I find something so satisfying about seeing my eval suites get saturated, it pushes me to find new ways to experiment and learn about models.
At the same time, I think the challenge with most benchmarks today, including my own, is that we tend to focus too much on trying to give models super hard tasks. Like I just said, a new set of models comes out, an eval suite saturates, and it's off to try to build a harder one.
But what is still missing is real signal around what models we all need for just normal daily coding tasks.
This was originally my goal with VulcanBench, then I got too interested in challenging the frontier and my core eval suite, now Frontier v4 became my focus.
Now I'm working on v5 because I'm addicted to challenging the frontier and really can't stop at this point...but I also want to get back to my original mission.
So I'm working on a new kind of benchmark, something I think is uniquely different from anything else out there.
The goal is not to challenge the frontier, because most of the daily coding work engineers are doing, isn't really that challenging to the frontier, it's normal daily work.
And I think many people are very lost now trying to figure out what model and effort level they need for regular daily work.
While model routers exist, they are in a very nascent stage today. I don't think there's any model router that has the sophistication to know, ah, this is a medium-sized Python repo, doing mostly Pandas stuff, let's use this model and effort level.
I just don't think current model routers would know the difference between this, and say, a medium-sized Python repo, mostly doing say supervised ML with SciKit Learn.
Instead, model routers today mostly just pick an orchestrator, executor, and code review model, that will be cheaper than someone using one model for all three. Sure, it's better than just using Opus 5.5 High in Claude Code for everything, but it's not actually optimized in any way to the codebase size, programming language, etc.
I really want to be able to help people understand. Okay, what model do I want to use for planning, execution, and code review for a repo of a specific size, with a specific programming language, and for certain types of tasks leveraging specific libraries.
This also means rethinking how to score benchmarks, because every frontier model should be able to pretty much ace most routine tasks. So instead we need to look at things like accuracy, cost, time, and code quality, all together vs. just looking at something like accuracy and cost-per-task on a pareto chart.
All of what I have shared above, will become the Routine v2 eval suite, along with a new benchmark/scoring system designed to provide signal on routine tasks, broken down by language and repo size.
My goal is that I can essentially create a system where you can select your codebase size and programming language, and rank the difficulty of the tasks you're doing, and get a model and effort level out of it.
For example, you might select:
Repo size: Medium
Programming language: Python
Difficulty: Easy
and it might recommend Opus 5.5 Medium for planning, GPT-6.1 Terra Medium for execution, and DeepSeek V4.1 Flash High for code review.
This would be very different than if you used most of the model routers out right now, they might never use GPT-6.1 Terra Medium. They might always use the same model at the execution layer, and yes, it's 30% or 40% cheaper than if you just used Opus 5.5 High for planning, execution, and code review, but it's not actually the optimal model combination.
As usual, I like getting things out of my head, and into the world, which I've done a lot on X this year and through it, I've received a ton of great feedback.
So if you read this far, first, common, it's a Sunday morning, what are you doing reading my insanely long tweet?
And second, what do you think? I'm always looking for feedback. Comment and let your voice be heard.
(just please, no ai slop comments, real people, with real thoughts only, all accounts just posting ai comments like, "the real win is" will be blocked)
2
373
The people asked for a Sonnet comparison, and the people get what they want.
Super interesting.
A number of people commented on my GPT-6.1 Terra benchmark this morning and suggested comparing it to Sonnet.
Soooo, thought I'd whip something up.
Here's a comparison of GPT 6.1 Terra to Claude Sonnet 5.5 across all effort levels.
And the data is pretty clear here, Terra is a lot cheaper than Sonnet, but not quite as accurate until you get to Max.
Traces for all 115 Terra runs, with every tool call:
github.com/morganlinton/vulc…
Leaderboard: vulcanbench.com/leaderboard.…



















