Autobench (@AutobenchAI) header
Autobench (@AutobenchAI) profile picture

The LLM evaluation engine powering AI benchmarks and routers. We use multi-LLM evaluation for accurate and unbiased evaluation of LLM quality, cost and speed.

Joined June 2025
How does Botscanner work?
2
91
What 1$ buys you? Depends who you ask
Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI). Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try). On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots). Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna. Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol). 1/7👇
1
2
135
An LLM router that does its job offers huge ROI to enterprises and developers. Finding 1 from our Router Benchmark: The prize is real. At 1 million questions a month: • Always GPT-5.6 Sol: ~$79,000 at 87.5% • Hindsight pick of the cheapest model still good enough: ~$1,800 at 100% Higher quality. About 98% less spend. That is what routing is supposed to collect. Luna already gets most of Sol’s quality for ~$4,300. The orange bar is still on the table.
Replying to @pwk
And yet the prize is real. The poor perforance of routers would not matter if there were nothing to collect. There is. This is the monthly model bill at one million questions, using this benchmark’s real prices. Always calling GPT-5.6 Sol: about $79,000, at an 87.5% best-answer rate. A router that, with hindsight, picked the cheapest model still good enough on each prompt: about $1,800, at 100%. In other words: Higher quality, about 98% less spend. That gap is the prize. Routing is supposed to collect it. At 85.3% best-answer rate, a mid-cost model like Luna gets you most of Sol’s quality for about $4,300. The orange bar is how much saving is still on the table (and there is an aditional +15% in quality). 2/14
2
72
This morning we published the AutoBench Router Benchmark. 7 major LLM routers. 17 settings. Long, open-ended prompts. Real billed cost. Result: always calling a mid-tier model (GPT-5.6 Luna) beat every router we tested on both quality and cost. No exceptions. The shaded zone is the prize: better than random, cheaper than random. Luna is already in it. Every router we tested is outside it. Read the paper: autobench.org/reports/do-llm… Access the data: huggingface.co/datasets/Auto…
Do LLM routers work? I have some good news for @AnthropicAI and @OpenAI: LLM routers suck. Routers are systems that, given a prompt, choose among a set of LLMs which is the best to answer. The goal is to identify the model that guarantees a high quality response at the lowest cost possible. The problem they promise to solve is simple: why call Gpt Astra on a prompt that a low-cost open source model like @deepseek_ai can answer just as well? A nightmare scenario for the big labs like OpenAi and Anthropic. Well, we applied the @AutobenchAI benchmarking methodology to 7 major routers. Result: always calling a mid-tier model like GPT-5.6 Luna beats every router we tested on both quality and cost. No exceptions. Check out the quality-vs-cost Pareto board. The light shaded zone is the bare minimum prize: better than a random pick and cheaper than a random pick. Luna is already sitting in it. Every router we tested is outside it. It's even worse. The routers that look cheap just collapsed onto one flash model and landed on the random line. The ones that look strong just bought an expensive model. 1/14 👇
1
51
Yesterday out. Today on Bot Scanner. As you can see below, o3 immediately stands out. Careful though. Despite the discount, it still costs a fair bit (not as o1, but still a lot). And, in fact, you need Tier 1 on Bot Scanner to use it (get it with a $10 credit purchase). 2/3
1
42
The news of the day is that @OpenAI has opened o3 access via API and lowered the costs by 80%. But where can I ever find o3 to use it? Where else but on Bot Scanner? 😎 👇 1/3
1
1
1
636
Just one week from launch, Bot Scanner enters the Amazon @AWSstartups Program. We are one of the 12 Italian startups in the AWS Startups Showcase. It's not just a nice recognition. It's also money ($) that is very useful to us (we spend a lot on AWS cloud infrastructure). If you are in the program, let us know and visit our showcase page. Still not using Bot Scanner? Our platform selects the best LLM answers for your every question? We still have codes available with $3 of free credit for those who want to test Bot Scanner. All you have to do is leave a comment here and follow our profile (so we can text you the code). We will send you the invite with the $3 of free credit code.
1
1
808