VulcanBench (@VulcanBench) header
VulcanBench (@VulcanBench) profile picture

Benchmarking models across effort levels and harnesses on eval suites with real coding tasks. Because you don't need High or Max effort as much as you think.

Lake Tahoe
Joined March 2020
Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)Photo by VulcanBench (@VulcanBench)
VulcanBench retweeted
Look at these results on @VulcanBench by @morganlinton on GPT-6.1 Sol are various effort levels. It's the only consistent benchmark doing the unglamorous work to give us this level of detailed insight that I'm aware of. 6.1 Sol at Low looks incredible, and if you combine it with multi-model code reviews for example (which compound engineering does), you can really get great results while pinching pennies here. I'm gonna try it on fast mode too.
Okay, this is definitely one of the most interesting benchmark comparisons I have ever done. GPT 5.6, 6.0, and 6.1 Sol, across all effort levels on @VulcanBench Frontier v4. And honestly, I don't think I even need to give any analysis or explanations because the results really do speak for themselves. But one thing I will highlight is that GPT-5.6 Sol and GPT-6.0 Sol all struggled at Low effort, GPT-6.1 is a beast at Low effort. I am now convinced, GPT-6.1 Sol is a very good model, and you can comfortable use it, at Low effort, and see strong, insanely cost-effective results. I don't say this often but I can tell you, after seeing this data, I'd be crazy not to make this one of my daily drivers. Full details from the benchmark now live on the site: vulcanbench.com/benchmarks/s… Live long and benchmark 🖖
6
3
23
3,105
VulcanBench retweeted
I am building a new eval suite at @VulcanBench that I think I'm going to call VulcanBench Frontier Systems v1. My idea is to build an eval suite around difficult, multi-hour engineering work in underrepresented systems, with mechanically verifiable outcomes. Looking for experts with experience in any of the areas below to help design and validate tasks: - Verilog/FPGA - Erlang - Ada/SPARK - COBOL/Fortran - Compilers Not looking for tricks, puzzles, etc. I really want tasks that represent real work, i.e. things like debugging, implementation, optimization, migration, or recovery in systems an expert actually cares about. If this is your domain, or you know someone who you think would be a good fit, I’d love an introduction. Oh and this is not a paid gig, I'm spending thousands of dollars a month running benchmarks and building my own eval suites. I have zero funding atm. Unlike Gartner, you can't pay to look good in my benchmarks 😜 You would get credit of course for your contribution, and know that you're helping to contribute to frontier model benchmarking. DMs open.
9
2
27
5,154
VulcanBench retweeted
Update on the Fable 5.1 all effort sweep on @VulcanBench with my new Coding Intelligence Index v4 eval suite. Low effort just finished, onto Medium, slowly but surely getting there. Very excited to share the results of this one. And in the process, retiring the leaderboard for Eval Suite 3 as I build a new leaderboard with this new eval suite. One of the challenges, and exciting things about benchmarking frontier models is that nothing is static, you have to always change and update, and evolve your eval suites and methodologies. Exciting journey ahead, and honored to be able to start this new eval suite with a model like Fable 5.1 🖖
5
1
30
2,168
VulcanBench retweeted
Ask yourself if you really need “high” or “xhigh”. Scores and prices from @VulcanBench vulcanbench.com/benchmarks/1…
Grok is so much smarter in its build harness than in the app it’s not even a comparison. The difference is so stark that I often go out of my way to open Grok in the CLI instead of using the app/web.
2
3
119
VulcanBench retweeted
So I'm thinking about benchmarking models outside of the API, in subscriptions instead. The reason being. It's getting pretty expensive to benchmark through the API, and I realized I can benchmark through subs, at a fraction of the cost, and still track tokens and calculate the costs if it were through the API, or through a third party harness that uses API+ pricing like Factory or Cursor. This would also make it easier for me to test across harnesses, which is something I have really wanted to do with @VulcanBench since I hear harnesses like Pi are performing really well, and I'd love to compare performance from harness to harness, but this would get sooooo crazy expensive if I did it through apis. Anyone see any issues with this or does this sound like a good plan? Could be the difference between spending thousands of dollars a month to run my benchmarks and spending hundreds. Which also means I could run even more benchmarks. But I care a lot about what the community thinks, and if there's anything that I'm not thinking of that could make this somehow unfair or not representative in any way? Please share your thoughts and let me know what you think, I'm doing free open source benchmarking for all of you more than anything else, so comment and let your voice be heard!
8
3
20
1,840
VulcanBench retweeted
In case you missed this yesterday. Here’s my table for model selection based on my latest benchmark run.
New VulcanBench results for GPT‑5.6 Terra and Luna are in 🔥 I tested both models on 23 difficult software-engineering tasks sourced from real merged OSS PRs, using Docker-sandboxed agents and fixed step and wall-clock budgets. Each model ran at low, medium, and high reasoning effort, with three passes per task: • 414 benchmark runs • 69 runs per model/effort cell • $67.76 in total API spend The best observed Terra result came at medium effort: • Low: 81%, $10.48 • Medium: 87%, $13.05 • High: 86%, $26.28 Luna improved steadily as reasoning effort increased: • Low: 77%, $2.10 • Medium: 80%, $3.90 • High: 86%, $11.95 Here's what was the most interesting to me, Luna at high effort matched Terra at high effort while costing less than half as much. Terra’s medium result was also slightly higher than More details in the pretty charts below. Live long and Benchmark 🖖
6
7
46
7,461
VulcanBench retweeted
Interesting. Qwen is the first model that it looks like I will have to provide ID Verification to call the api. Didn't need this for DeepSeek. Curious if anyone knows if there is a way around this?
10
3
22
2,397
VulcanBench retweeted
So I decided to try something interesting last night. Since DeepSeek and Grok are so cost-effective, I can afford to run them with 3 passes on @VulcanBench. When I did, the rankings changed. DeepSeek V4-Flash actually dropped in the rankings by two and Grok 4.5 High took the #1 spot. My challenge now is, while I'd love to run all the other models at 3 passes, this would cost over $350, and VulcanBench is just a little self-funded project of mine. For now I've added whiskers to the chart and a footnote to make it clear which models have been run @ 1 pass vs. @ 3 passes.
27
8
5
152
19,666
VulcanBench retweeted
Try Grok 4.5!
BREAKING: Grok 4.5 leads VulcanBench’s new coding benchmark. 🔥 Grok scored 91.3%, solving 21 of 23 real-world software tasks across five languages, beating Claude Fable 5 and GPT-5.6 Sol while also owning the cost-efficiency frontier. Grok keeps winning. 🏆
1,417
1,011
79
6,396
3,122,284
VulcanBench retweeted
Pretty amazing, grok wins on cost AND accuracy!
Woke up to a finished benchmark, and a very interesting outcome. Grok 4.5 beat Fable 5 and GPT 5.6 Sol in both accuracy and cost 😮
1
1
20
2,252
VulcanBench retweeted
The minimum to run the model is ~$20K in hardware and you get ~20 tok/s out ~$20K gets you around 34.6B tokens at a 12:1 input to output ratio assuming good token caching If you ran the hardware 24/7, it would take roughly 5.5 years to break even
GLM 5.2 costs $1.40/4.40 per Mtok at 40 tok/sec and people seriously consider buying GPU rigs for it
124
56
31
1,447
338,567
VulcanBench retweeted
GLM-5.2 scores pretty well with max reasoning actually beating GPT-5.5-low and Opus 4.8 low on DeepSWE but it still needs to work on reasoning efficiency
GLM 5.2 is now on DeepSWE as the top open-source model on our leaderboard. With a pass@1 score of 44% at max effort, GLM 5.2 is indisputable #1 open-source model besting Kimi K2.7 Code by 17%.
22
8
264
32,324
VulcanBench retweeted
Okay, shipped. VulcanBench v0.1 is here. Very early beta, expect bugs, expect things to break. When they do, either report them to me, or put in a PR and fix em' yourself. The journey begins. It's time to bring more transparency to llm benchmarking. Live long and Benchmark 🖖
Okay, it's official, VulcanBench v0.1 is here. Free and open source llm benchmarking tool, pre-loaded with 52 tests so you can really go deep. The goal is simple: completely transparent benchmarking, every detail from the code that runs the tests, to the tests themselves, is available, for free, for everyone to see. Please test, test, and test some more, and give feedback, or even better, submit PRs to make it better. Live long and Benchmark 🖖
5
4
1
24
3,323
VulcanBench retweeted
Okay, so I've come to the conclusion that I need a 3090, like yesterday. Really want to run more powerful LLMs locally at home. Relatively new territory for me, I'm far from an expert, so was chatting with Perplexity about it. Here's what it thinks I need, hoping someone like @0xSero or @LottoLabs and let me know how right, or wrong it is, and what I really need. Trying not to break the bank so I'm okay starting small(ish) 🤏
34
2
58
8,117
Hey Game Devs! 🌸 I'm Lya a cozy game dev, my objective is to try out as many demo as possible during the Steam Next Fest! If your indie game is participating and you want a peer feedback, don't hesitate to drop your demo link bellow ✨🚀 #SteamNextFest #NextFest #indiegames
65
11
112
4,837
It's time for the final episode of 5 Things I Bring to GDC! Alexa Ray Corriea reminds us that one of the most important things at GDC is to stay hydrated and believe in yourself. 3 weeks left before #GDC2026 what are you bringing? bit.ly/48w1DxK
1
6
1,226
VulcanBench retweeted
Starting a more meaty project this weekend - something I've dreamed of doing, but really didn't have the bandwidth to do until now. With Opus 4.6 and GPT 5.3 Codex, I decided, it's time to do it. I'm going to build a game engine. But a kinda different one, a game engine for building CLI games. It will be Open Source, and I have no idea how long it's going to take to build. But the first version just passed a crazy set of test suites that Codex put together. We all spend a ton of time in the terminal these days, I'm hoping to help us have some of that time be spent just having fun, and playing games. Introducing Terminal Forge. More to come.
16
3
1
70
4,711
VulcanBench retweeted
This game looks insanely cool.
1
1
6
971
VulcanBench retweeted
this was one of the biggest weeks in AI because claude opus 4.6 and gpt-5.3 codex dropped basically at the SAME time. they solve the same problem in VERY different ways. - opus spins up agent teams and disappears for a while. - codex stays with you and ships ridiculously fast. i tested them both for you in this episode of @startupideaspod (on yt/spot/apple/X) with my pal @morganlinton we set them up live, tuned them properly, and had each one build a polymarket clone from scratch. you’ll see where each one shines, where it struggles, and which approach actually fits how you work. if you’re trying to decide what to use next, this episode will save you a lot of trial and error.
90
81
9
753
92,434
Weight decay is usually presented as “encouraging simpler solutions”, but I tend to think that the real benefit is the soft pruning of noisy / unhelpful features. Without decay, a weight can random-walk to a large value even if the input is completely random. Momentum and Adam/rmsprop gradient normalization also reduce the impact of noisy features — they won’t shrink a weight with a noisy input, but weights with less noisy features will move relatively faster than with plain SGD.
32
9
3
404
57,124