Autobench retweeted
A few weeks ago we said LLM routers don't route. Then @OpenRouter shipped a new one.
On 6 November they launched a rebuilt Auto Router. So our team @AutobenchAI ran it on the exact questions from our October router benchmark. Not what you would expect. At its top setting its Best Answering Ratio went from 79.1% to 65.2%. A random pick scores 64.7%.
We also tested two new routers, @coniferbuild (a winner) and @daridotdev.
Here details of what we found:
1. The new OpenRouter Auto Router at its best setting scores about the same as picking a model at random (65.2% vs 64.7%). Its other three settings score below random. It agrees with the old router on only 1–12% of picks. Its top picks are @Nvidia Nemotron Ultra, @MiniMax_AI M3 and @deepseek_ai V4 Flash. At its highest-quality setting it never picks @OpenAI GPT-5.6 Luna, the best-value model on our board.
2. Conifer does route. It beats a random pick from its own models by almost 15 points. It still loses to always calling Luna (lower quality, higher bill), but it is by far the top performer among all routers we tested (except for our own Autobench router).
3. Dari scores 79.1%, the same as the old OpenRouter Auto Router. But it only chooses among 8 strong models. A random pick from those 8 already scores 73.9%.
So the October result stands: no commercial router we tested beats calling one mid-priced model... Well, no router except our own. The AutoBench Router at Balanced scores 87.8%, above always calling GPT-5.6 Sol (87.5%), at 8% of Sol's bill. Try it and get $1 free credits.
Links are in the replies.
1/3👇
A router is measured if three things can be checked by someone outside the vendor:
Who graded it
What it was graded on
What the buyer can verify on their own traffic.
Botscanner.ai checks all three
3
17
"Which model is best?" and "which model is enough for this task?" are different questions.
We test the second one, on your own prompts
2
40
Autobench retweeted
Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).
Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).
On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).
Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.
Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).
1/7👇
We audited seven routers and none beat always calling one mid-priced model. So we built one: three models, three settings, $1 free credit. Apply for early access to the beta
botscanner.ai/
3
80
Autobench retweeted
There’s significant efficiency to be gained by switching from the large frontier models of @AnthropicAI and @OpenAI to cheaper Chinese alternatives like @Zai_org GLM. However, this approach has its own drawbacks:
1. Models like GLM handle a wide range of use cases well, but they still fail on a small percentage of cases (roughly 1–5%) compared to SOTA models. These failures tend to hit exactly where it hurts most—the more complex, high-stakes requests.
2. Competition among Chinese labs is just as intense as among US labs. This drives continuous improvement, which means companies will likely need to keep switching to newer, more advanced GLM versions (or even different model families). Managing that transition yourself is painful and operationally heavy, especially if you’re managing your own hardware infra.
Taken together, this strengthens the case for LLM routers that are continuously upgraded to the latest models and can intelligently route real-time user requests, including the long-tail complex cases that still need more powerful (and expensive) models.
That’s exactly what we’re doing at @AutobenchAI with the router we released last week. It delivers 90%+ savings versus SOTA models while still giving you full SOTA capability on the ~5% of cases that actually need it. Try it out.
Revolut: Europe's most valuable startup ($200B) just showed the world its AI strategy.
Every company will end up copying it.
At the start of this year, Revolut leaned on the frontier labs. Its 2,000 engineers used Claude Code and Codex, and some burned $1,000 to $10,000 a month each. The bills kept climbing, and the returns weren't there.
So Revolut switched its default to an open-source model called GLM. All it needs is GPUs, which it rents from Together AI.
The results, by Revolut's own count:
- The bill is about a tenth of what it paid OpenAI and Anthropic
- Engineers get unlimited tokens instead of spending caps
- Coding time is down 60% and lead time is down 70%
That's the chart below.
Revolut still uses Claude and GPT, but only as "the exception," for the tasks they do best. Everything else runs on open source.
So here's the question I keep coming back to: how many use cases can companies really justify sending to frontier models?
My answer is very few. Plenty of companies still route everything to OpenAI and Anthropic today. For how long?
I think open source ends up handling the vast majority of corporate AI work, and frontier models get a short list of the hardest jobs. That's a problem for labs whose revenue forecasts assume unlimited growth.
If you appreciate unpopular takes, follow me @m0xt_ for more.
The prize is real. A per-prompt policy that picks the cheapest model still within the quality bar scores 100% Delta-Hit at $0.0018 per question.
Always-Luna: 85.3% at $0.0043.
That is roughly 12 quality points at about 98% lower cost. No router collected it, until we launched botscanner.ai
huggingface.co/blog/PeterKru…
1
53
What 1$ buys you? Depends who you ask
Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).
Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).
On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).
Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.
Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).
1/7👇
Do routers actually route? Not really. We tested the most used routers and 5/17 commercial settings sent at least 90% of prompts to one model, so their quality number was that model's number.
Why pay for something you can do it yourself? Try botscanner.ai, the only router that actually routes
Replying to @pwk
One of the major failures we pointed out in our router benchmark is that most commercial and open routers don't actually route.
Five of the 17 settings sent 90%+ of prompts to one model. Turn the knob from cheap to quality and you often just got a different constant.
The @AutobenchAI Router does not do that as you can see in the graph. Prompts are routed between the different models, with the mid-tier model taking most of the calls (note that "Cost" routes only between two models).
If a vendor will not show you that mix, you are paying for a bookmark.
4/7
2
67
"LLM Routers are threatening the Frontier Labs' revenues". Another great article about the launch of our router.
Remember: you can try our router yourself. It's easy and you get $1 free credit.
startmag.it/innovazione/i-ro…
We're in the media🎖️
Our LLM Router on the media. Nice article by @Startupbusiness that explains the benefits of the @AutobenchAI routing technology: get Frontier model quality in the answers to your prompts at less than 10% the cost.
startupbusiness.it/quale-llm…
(Article in Italian)
1
30
Today we are releasing our LLM Router that gets you higher quality than Gpt-5.6 Sol at less than 10% of its bill. It also beats by far all major commercial and open-source routers on the Autobench Router benchmark.
We are also handing $1 free credits to test it on our user-facing service: botscanner.ai. For custom routers, reach out directly.
Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).
Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).
On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).
Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.
Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).
1/7👇
4
115
The only reason to pay for a router is cheap answers that are still good. Nobody we tested delivers that.
Finding 5 from our Router Benchmark: The cell that pays for the product stays empty.
Every answer is a hit or a miss, and cheap or costly. Cheap + good is the product. It stays a minority for everyone.
Strong routers buy expensive hits. Cheap routers buy cheap misses.
The prize is real. Nobody is collecting it.
Replying to @pwk
If they don’t agree on the model, they also don’t land where a router would pay for itself.
Every answer is a hit or a miss, and cheap or costly. The only cell that justifies paying for a router is cheap + good.
It stays a minority for everyone.
Strong routers mostly buy expensive hits. Cheap routers mostly buy cheap misses. Nobody fills the cell that would make the product worth it.
7/14
1
35
If there were a right model for each prompt, independent routers would find it. They don’t.
Finding 4 from our Router Benchmark: Median agreement is zero.
Across 59 cross-vendor pairs, most pairs never pick the same model.
The bright squares are the exception: both sides collapsed onto the same cheap default. Everything else is dark.
Replying to @pwk
Maybe they just disagree because each one found a different right model?
If there were a discoverable right model for each prompt, independent routers would converge on it. They don’t.
Across 59 cross-vendor pairs, the median agreement rate is zero. Most pairs never pick the same model.
The bright squares are the exception that proves the rule: both sides collapsed onto the same cheap default. Orca’s quality setting and OpenRouter Auto-low agree on almost every prompt for that reason. Everything else is dark.
6/14
1
33
If a router sends 90% of your prompts to one model, you are not routing. You are paying for a bookmark.
Finding 3 from our Router Benchmark: Most of them aren’t even trying to route.
Five of the 17 settings send 90%+ of prompts to a single model. At that point the quality number is just that model’s number. It is a knob that selects a constant.
Turn the knob on Orca or OpenRouter Auto and you mostly get a different default.
Replying to @pwk
Look at the decisions and it gets clearer why. Most of them aren’t even trying to route.
Five of the 17 settings send 90%+ of prompts to a single model. At that point the quality number is just that model’s number. It is not a router. It is a knob that selects a constant.
Turn the knob on Orca or @OpenRouter Auto and you mostly just get a different constant. Cheap mode, quality mode — same trick, different default.
4/14
1
32
Autobench retweeted
Most commercial routers promise to cut your AI bill by 60%, 70%, even 80%. They don’t.
Finding 2 from our Router Benchmark: None of them beat a mid-tier model.
A router earns its keep only if it beats always calling Luna: better answers, and a lower bill. You do not need a router just to enter the prize zone. Luna is already there.
We checked 17 settings across 7 systems. None did both.
We bootstrapped the board 10,000 times. The chance any of them ever does is zero.
Replying to @pwk
Most commercial routers claim they will save you 60%, 70%, even 80% of your bill. They don't.
We checked 17 settings across those 7 systems. A real router earns its keep only if it beats always calling Luna: better answers, and a lower bill. You do not need a router just to enter the prize zone. Luna is already there.
We bootstrapped the data 10,000 times. The chance any of them ever does is zero.
3/14
An LLM router that does its job offers huge ROI to enterprises and developers.
Finding 1 from our Router Benchmark: The prize is real.
At 1 million questions a month:
• Always GPT-5.6 Sol: ~$79,000 at 87.5%
• Hindsight pick of the cheapest model still good enough: ~$1,800 at 100%
Higher quality. About 98% less spend. That is what routing is supposed to collect.
Luna already gets most of Sol’s quality for ~$4,300. The orange bar is still on the table.
Replying to @pwk
And yet the prize is real. The poor perforance of routers would not matter if there were nothing to collect.
There is.
This is the monthly model bill at one million questions, using this benchmark’s real prices.
Always calling GPT-5.6 Sol: about $79,000, at an 87.5% best-answer rate.
A router that, with hindsight, picked the cheapest model still good enough on each prompt: about $1,800, at 100%.
In other words: Higher quality, about 98% less spend. That gap is the prize. Routing is supposed to collect it.
At 85.3% best-answer rate, a mid-cost model like Luna gets you most of Sol’s quality for about $4,300. The orange bar is how much saving is still on the table (and there is an aditional +15% in quality).
2/14
2
72
This morning we published the AutoBench Router Benchmark.
7 major LLM routers. 17 settings. Long, open-ended prompts. Real billed cost.
Result: always calling a mid-tier model (GPT-5.6 Luna) beat every router we tested on both quality and cost. No exceptions.
The shaded zone is the prize: better than random, cheaper than random. Luna is already in it. Every router we tested is outside it.
Read the paper: autobench.org/reports/do-llm…
Access the data: huggingface.co/datasets/Auto…
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck.
Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic.
Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions.
Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick.
Luna is already sitting in it. Every router we tested is outside it.
It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model.
1/14 👇
1
51
Announcing our LLM Router Benchmark.
We audited 7 LLM routers across 17 settings on a fixed benchmark, with rules locked before any router was called.
None beat one mid-priced model like Gpt 5.6 Luna on quality and cost together. Not once in 10,000 bootstrap resamples.
The perfect router would cost 98% less than always calling the top model. Nobody has collected it yet.
But this is just half the news. More on Thursday
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck.
Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic.
Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions.
Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick.
Luna is already sitting in it. Every router we tested is outside it.
It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model.
1/14 👇
6
231


















