Gemini 4 Neon has the same affliction, doing terribly on Runescape Bench because it tries super-ambitious strategies yet has terrible time management and doesn't finish them before the buzzer
The strongest models check the clock ~every 30 seconds. Neon checks it 6x less often AND chooses long complex strategies.
1
5
546
Cool results, especially enjoyed the human vs AI PvP scenarios, where thankfully @SethBling could outmaneuver and outsmart agents
Replying to @kradleai @SethBling
We’ve designed kradle.ai, a bench that’s messy, unpredictable, hard to saturate, and full of others (AIs & humans) pursuing their own goals. Like the real world.
You may be thinking… Claude?!? Carbon?
OpenAI takes so many top slots because their models are more iterative and adapt to their environment. Astra 6.2 iterates on its approach to tasks ~4x faster than Fable 5.2.
In general, Claude models tend to think for longer and then stick with the strategy they come up with.
And, Carbon is #16 due to latency.
Astra and Sol 6.2 are new leaders on Runescape bench.
Their signature strategy was completing 2 different quests and collecting both rewards simultaneously, beating records I'd thought were saturated.
max retweeted
for whatever amount of compute you have, you will get much more research done if you can just make your experiments smaller
now featuring 2x more images, text/image search, and a faster index, browse @AREdotNA slipperily
13
724
max retweeted
Hy4 High inside Pi is (unofficially) is a Pareto point in RuneBench with 6.23 score at about 0.59 USD/run
Hy4 High on OpenCode 1.19 scores 4.92 at 0.36 USD/run
(curious now to run DS 1017 on DSH minimal)
I can share the data if you want to officially validate @maxbittker
Gemini 4 Carbon underperformed on Runebench.
Interestingly, it attempted ambitious strategies (quests) then failed to complete 7 of them, wasting more time than any other model and hurting its score.
There's a youtuber producing an incredible educational series about learning advanced agent techniques via runescape, so awesome to see!
This second video is more advanced and covers remaining skeptical, building harnesses and visualizations, saving on token usage via model routing. Really great stuff
youtube.com/watch?v=Rnf5csei…
13
960
max retweeted
Fable 5.2 is very good at painting!! 🎨 Congratulations to the @claudeai team!(Video + Audio made in code).
Comparison below between Fable 5.2, Fable 5.1 and Gemini 4 Argon on the same prompts. ⬇️
At default effort, it's VERY similar profile to Opus 5.5, but at higher thinking it pulls ahead and is on the pareto frontier
maxbittker.github.io/runeben…
12
2,604


















