Open source agentic coding harness where local and frontier models live in harmony.
Joined January 2009
- Tweets4.5K
- Following340
- Followers4K
- Likes7.3K
Albatross retweeted
I am going to start to share the leaderboard at VulcanBench more regularly.
VulcanBench is the only benchmark I know that benchmarks across effort levels, on an eval suite of 100% real engineering tasks.
Getting ready to sunset eval suite 3 and eval suite 4 is almost ready to go.
Here's the current leaderboard for Eval Suite 3, Grok 4.5 High is the highest scoring model still with Fable 5 Low, yes Low, right behind it.
What I've been able to uncover with VulcanBench is that way too many people are running models at Max effort, thinking that buys them more accuracy, when really it just costs more and uses more tokens so takes more time.
If you're still in the mode of, tell my agent to do something then go get coffee, you're probably still living in the past. You can move faster, with higher accuracy, the key is not thinking you need Max effort all the time.
Time to share the leaderboard for VulcanBench, Eval Suite 3.
As a reminder, these are 100% real engineering tasks, from real repos. No puzzles, no math problems, no exotic architecture.
Our leaderboard takes the data we have from our full effort sweeps, every effort level, and reports the effort level that gave the highest accuracy.
Right now, so many engineering teams just run a model on High or Max, assuming that's what they need to get the best accuracy.
But for most regular, routine, engineering tasks, the stuff we do day in and day out, that's overkill. And it means the company you work for is paying a LOT more money for tokens, all to get the same accuracy, just a lot slower.
Currently Grok 4.5 High still tops the chart at 90, with Fable 5 Low right behind it at 89 and DeepSeek V4-Flash Max right behind that at 88.
If you're using Grok 4.6, Medium is actually the sweet spot for effort, and Opus 5, which everyone seems to be complaining about these days...well you're probably using it on High, which actually doesn't perform as well as low.
Effort matters.
Live long and benchmark 馃枛
