38 of 38 tests passed, and the app would not start. The jar had no Main-Class.
An AI reviewer with no access to the build's notes caught it by running java -jar, the way a user would. Its score only counted with proof attached, so it had to launch the app, and the tests never did.
We compared six sets of October runs, same spec and same model, with and without that review loop. Four came out more complete, judged by how much of the project shipped, and two came out less complete. The median run with the loop took about 3.4 times as long. That is the trade: it catches the app that passes its tests and does not start, and you wait more than three times as long for it.
This is @mattshumer_'s Gauntlet Loop, written up in July.
Ask a coding agent if it's done and it says yes.
Favur Evals no longer takes its word for it. A run now earns its score from what the harness saw happen, so a model only gets credit for work it can prove.
evals.favur.dev/go/x/rubric-…
Favur Evals - weekly recap.
This week a DeepSeek model wrote the best test suite we have ever graded, a perfect 100 out of 100 for a C++ arena game that five other models coded, seven points clear of the next best suite on our board. @deepseek_ai 's V4.1 Flash did the testing; the rest of the team wrote the game it checked.
Also this week, @Kimi_Moonshot 's Kimi K2.6 built 98 percent of a small orbit-animation app's spec, the most of any Python run of that app, and @NVIDIAAI 's Nemotron 3.5 Super VL streamed its output about three times faster than the typical run on the board.
new recap every Friday ↓
evals.favur.dev/go/x/evals-r…
"Stop re-reading files and re-running commands you have already run. You already have what you need."
Our harness sent that to 21 agents in one run, each after 15 cycles stuck in one step. For 17 of them it was the only warning.
Across that 174-minute run, the harness posted 607 messages as if the user had typed them. A person typed none of them.
Which message do you retype to your agent the most?
Favur Evals - weekly recap.
We rescored every run this week. A quarter of each score is now whether the final build was verified complete, so your model's rank may have moved.
Same task for every model: build one window with a circle orbiting a circle.
@StepFun_ai 's Step 5 Preview finished more of the Python build than any model we have run on it.
@OpenAI 's GPT-6.1 Sol wrote the highest-scoring TypeScript test suite, with every test passing and every line covered.
@Zai_org 's GLM 5.3 Prime built it in Java, C#, Rust, Go and JavaScript, and in each language it ranks first or second for fewest tokens and turns.
new recap every Friday ↓
evals.favur.dev/go/x/evals-r…
Our agents spent 296 wall-clock hours building a WebGL renderer with no GPU under it. It draws a triangle, and it is not finished.
The costliest mistake was one log line. The agents asked the human operator whether to keep chasing a goal they could not reach. An automated reply came back, and the log recorded it as the operator's decision. From there the plan aimed at 80% and 50% of the conformance tests, down from 95% and 90%.
This is the postmortem we promised on our @terminalbench wasm-render attempt. It shows each place a check passed and proved less than it seemed to.
We retry the challenge in a few days. Follow us to see how the second attempt goes.
dev.to/distilled/post-mortem…
The Gauntlet Loop from @mattshumer_ runs as a prompt. We build a harness, so we wrote it into the code.
In Claude Code and Codex, that prompt catches what a builder's own review lets through. In Favur it is now code: when the agents say a phase is done, critics who never saw it built launch the app, drive it, and run the tests. Each one is a different expert, like a QA engineer, a security engineer or a first-time user, so each looks for something the others miss.
Nothing passes on the agents' word. A critic's score counts only with proof behind it, like a screenshot of the running app or the output of a test. Below the bar, fixes go in and new critics check again, and if the bar is still missed at the last round, a person decides.
Some results from one public run: a QA critic showed that deleting the speed factor from the motion code would leave every test passing. Next round, a new critic found NaN and Infinity going straight into the math. By round three the build rejected them, and when a fresh panel of critics scored it again, the lowest score rose from 6 to 7 out of 10, the bar we set. Our September devlog is the video below, and the run page is in the reply.
Favur Evals - weekly recap.
Same app, nine languages. The app is one window, a circle orbiting a circle, and the spacebar reverses it. This week @Zai_org 's GLM 5.3 built it in JavaScript, C#, Rust, Java, C, Python and Go. Find your language on the board.
@AnthropicAI 's Claude Sonnet 5.5 wrote the highest-scoring test suite we have measured on this job, on its first run, and its tests execute every line of the code.
@OpenAI 's GPT-5.6 Sol made over 4,000 requests without one failing, the longest clean build on the job.
@XiaomiMiMo 's MiMo-V2.6 Pro Ultraspeed generated tokens faster than any other model we have run on this job.
new recap every Friday ↓
evals.favur.dev/go/x/evals-r…
Models have seen read_file(path=...) so many times that the name brings the parameter with it. Ship a read_file whose argument is file_to_read and the model keeps sending path anyway.
We hit the same thing with commands. Our command tool takes the program, its switches and its paths as separate fields, not one shell string, so we named it execute_structured_command. Call it execute_command and the model hands it a shell string. The old execute_command is still there, hidden from the tool list but callable, for models that reach for it out of habit.
Favur Evals - weekly recap.
On our @terminalbench wasm-render attempt this week, the agent that reviews each sprint caught three tasks marked done that pointed at commits that did not exist. They got built for real and re-reviewed before the sprint closed.
@Zai_org 's GLM 5.3 FlashX posted the only perfect deliverables score on our solar-system job, and topped that job on process discipline (commands that succeed, frequent checks of its own work), just ahead of the full GLM 5.3.
@OpenAI 's GPT-5.6 Luna now scores highest of any model in our code-review role.
@AIatMeta 's Muse Spark 1.3 Contributor now scores highest in our orchestrator role, the agent that manages the whole build and hands each sprint to the rest of the team.
@spacexai 's Grok 4.7 turned in the sixth-best overall run on the solar-system job, in its first run there.
new recap every Friday ↓
evals.favur.dev/go/x/evals-r…
Favur Evals - weekly recap.
@_inception_ai 's Mercury 2.5 led a run on our circles app - one window, a circle orbiting a circle, spacebar reverses it - and is the only run ever to take full marks on test quality there: deep assertions, dense against the source, fixtures and parametrization throughout.
@spacexai 's Grok 4.6 led a run that almost never sent a malformed, retried or out-of-bounds tool call - only two runs have ever been cleaner on that job.
@OpenAI 's GPT Terra 5.6 now tops four of the fourteen agent roles we score separately - code, spec, pseudocode and platform - more than any other model on the board.
new recap every Friday ↓
evals.favur.dev/go/x/evals-r…
A cost breakdown from @galdawave put 84% of an agent's API bill on cache reads and writes, not on the tokens everyone watches. One line in it is almost an aside: the numbers jumped again one request after compaction.
A prompt cache matches on an exact prefix of bytes. Compaction rewrites the earliest turns, which is the exact span a cached prefix covers, so that jump is the cache working, and it is the clearest drift signal you get. What you pay is set by where things sit in the prompt, not by how much you send.
Sorting a prompt by what changes per request is the easy half. Holding the boundary is the work, because when someone later tucks a timestamp into the top half nothing fails, nothing goes red, and the bill just goes up.
Ours holds it in the code rather than in anyone's memory. The builder for the cacheable half accepts no per-cycle arguments at all, so it cannot vary between cycles, and a test renders it twice and compares bytes to prove it still does not. That test has one requirement that is easy to miss - its builder must not accept per-request data, or it can be handed the same value twice and pass green while production varies it every call.
The honest limits of ours, and five steps for your own harness ↓
dev.to/favur/your-cache-bill…
84% of our Anthropic API cost came from cache reads and writes.
Not input tokens, not output tokens.
Cache.
At Google, I honestly never thought much about the cost of AI. Joining an early stage startup completely changed that. Every dollar matters, so I became pretty obsessed with understanding where our token usage was actually going.
Most of our usage made sense. Our team averages around 70 to 100 PRs a day, so a large token bill is expected.
Then Fable was released and something felt very wrong. We were producing fewer PRs, but our cost was nearly 20x higher, and yes, I know what you’re thinking. Fable 5 is much more expensive than opus, but even after accounting for that, the math still didn’t work.
So I spent a week tracking everything claude code reported and comparing it against the Anthropic platform. The numbers mostly matched.
The surprising part was where the money was going. 84% of our usage came from cache reads and writes, split almost evenly, with cache reads slightly ahead.
In theory, caching should make things cheaper. In long running sessions, especially when claude code is acting as the main orchestrator, it became the biggest part of our bill.
Compaction helped, but only briefly. One request later, cache reads and writes would jump again. So I changed the workflow.
I now use Fable for planning, create the Linear ticket, then start a completely fresh session with Sonnet and let Codex handle the implementation. That simple change reduced our cost by around 50%.
Biggest takeaway:
If you’re paying for API usage, don’t just watch input and output tokens. Watch cache reads and writes very carefully.
Honestly, I suspect something similar happens on Claude subscriptions too. You just never see the bill.
Does a smarter model make your harness code obsolete? We held one harness fixed across five consecutive generations of @GoogleDeepMind's Gemini Flash, ran the same statement of work through each, and watched one number: calls the model invented that the harness could not execute, with the wrong shape or arguments that do not exist.
Flash 2.5 malformed about one call in fifty. Flash 3.7 malformed about one in three hundred.
The direction is real. Code that catches a bad tool call does less work on a new model than it did on an old one.
You still cannot delete it. The curve is not a line: Flash 3 preview was worse than the 2.5 it replaced, and 3.5 and 3.6 crept before 3.7 dropped. Retire a defense on the trend and you are covered for every generation except the one that backslides.
All five generations, the full comparison, and the raw logs:
dev.to/favur/gemini-models-r…
Gemini models really are getting better, and we ran the experiment that proves it. Five consecutive @GoogleDeepMind Flash generations, one coding spec, one harness, nothing changed but the model.
The oldest wrote out its reasoning on 2,596 of its 2,717 replies. The newest wrote it out on 31 of 461. That quiet run is the one that finished the same job in 43 minutes, against a 1.9 hour median for this job. It shipped a suite covering 91% of its code, where the norm is 68%.
The last year of this field would predict the opposite. The story has been test-time compute, models that get further by deliberating longer. Here is a line of five releases where visible deliberation falls away and the work gets better underneath it.
The same shrinking output shows up in the tool calls, which is where it turns into time. A call that does not match the tool's schema executes nothing, and the round trip is spent. Here is the share of all tool calls that came back schema-invalid:
2.5 Flash 1.91%
3 preview 2.15%
3.5 Flash 1.10%
3.6 Flash 1.02%
3.7 Flash 0.32%
A typical run of this job carries 1.42%. In counts, that is 75 bad calls out of 3,928 at the old end and 2 out of 631 at the new one.
Two of those five are printed on the runs' own pages, one flagged high and one flagged 4.4x under the norm. The other three we counted from the runs' telemetry, a second instrument, by a method that reproduces every published figure exactly.
Favur Evals - weekly recap.
@GoogleDeepMind 's Gemini 3.7 Flash wrote the third-cleanest code we have ever recorded on the circles job.
@Zai_org 's GLM 5.3 finished that same job with fewer turns and less output than almost anything we have run on it. Only one run has ever been leaner.
@deepseek_ai 's V4 Flash turned in the fifth-best test suite we have scored on it.
new recap every Friday ↓
evals.favur.dev/go/x/evals-r…
Most fixes for context compaction try to make the eviction judgment sharper. A pruning tool for the model, a smaller model scoring relevance, layered summaries.
All of them run that judgment at the one moment the system knows least. By then it is four hundred messages of undifferentiated text, and length and recency are the only properties left, both anti-correlated with what matters.
Favur is our team of agents that builds software from a written specification, and in it every message gets a retention class at append time, chosen by the call site that knows why the message exists. Forgettable, fuzzy, strict. Compaction reads the classification, never the content, and summarizes a class one tier before it drops it.
The trade is that a compaction routine is wrong in one findable place, and retention classes are wrong at whichever call site forgot. That failure is silent by construction.
The tiers, the thresholds, and what the design costs:
dev.to/favur/deciding-what-a…
Run the same statement of work through enough models and one pattern holds: no single model wins everything. On our circles board one model writes the cleanest code, another tests better, another finishes in half the time. @OpenRouter's new Auto Router goes straight at that problem - it picks the model per request, based on what the market actually uses for each kind of task.
So we handed it the whole team. Every Favur agent on one slug, openrouter/auto, router's choice on every request. We changed nothing in the harness - it dropped in like any other model - and because the routing follows the market, the same slug gets better as models do.
The score is in: 75.6 composite, above the median of our last ten circles runs at under a third of their median cost, and the top process-discipline score of the whole set. First run, day one of the new router. evals.favur.dev/runs/circles…
We ran our usual 2048 statement of work through Favur again, and this time the agents were told to generate their own art.
Xiaomi's @XiaomiMiMo v2.5 and v2.5 Pro split fifteen roles between them and shipped Monster Kitchen 2048 in 3h 45m.
Nobody was in the loop to fix anything along the way, so the visual bugs are part of the model's output too.
Scored run:
evals.favur.dev/runs/the2048…
Favur Evals — weekly recap.
four new models ran this week. @XiaomiMiMo 's MiMo v2.5 posted the board's highest single-model score, 81.2 on circles, and leads solarSystem too.
the only run above it split roles across two models.
new recap every Friday ↓
evals.favur.dev/go/x/evals-r…








