most AI products fail retention because they're instrumented like websites, not like decisions.
you can track clicks. you can't track whether the output was actually good.
the eval problem is also the product analytics problem. most teams don't realize it until churn starts.
Harness² makes the setup the learnable artifact instead of the weights. most teams have been treating the harness as config. it might actually be the thing worth versioning and optimizing.
there are three gaps in software quality: it works here, it works in tests, it works for users.
CI closes the first. evals close the second.
most teams building AI products have one without the other.
an instruction file for an agent and an eval suite for an agent are solving the same problem from different directions.
the instruction file says what you want. the eval suite shows you whether you got it.
most projects have one without the other.
most AI demos show the happy path.
the agent finds the file. the API responds. the user says what you expected.
real production adds: ambiguous requests, rate limits, stale data, partial failures, users who change their mind.
the demo is 10% of the work.
most AI agents fail on underspecified tasks, not wrong models.
"summarize this document" is not a task.
"extract the three decisions that need sign-off and the deadlines attached to each" is a task.
specificity is what makes an agent actually useful.
the difference between a benchmark and a capability:
a benchmark tells you what a model scores on a fixed test.
a capability tells you what it can do in the messy conditions of your actual use case.
those two things can diverge a lot. and they usually do.
a spec that your agent keeps failing is usually telling you something about the spec, not the agent.
most failed agent deployments are unclear requirements shipped as system prompts.
AI agent cost is almost always a retrieval problem in disguise.
the model call isn't expensive. sending it 40k tokens of context on every turn because you haven't built a memory layer is.
first thing to audit isn't the model. it's what you're loading into every prompt.
most AI cost overruns follow the same pattern: someone used a frontier model for a task that needed a confidence score. it's not a model problem. it's a design problem.
the architecture review that asks "do you actually need generation here" doesn't exist yet at most companies.
the scariest agent failures aren't wrong answers. they're confident wrong answers in the middle of a 20-step workflow where nothing looks broken until step 19.
the agent memory problem has two parts that usually get conflated. storage: what do you keep? purging: what do you discard, and when? most systems get the first part wrong. very few have even thought about the second.
most agent skills in public marketplaces have never been tested. someone wrote what felt right, shipped it, and moved on. the skill might help, hurt, or do nothing. nobody knows.
skills are prompts. prompts need evals. same standard as a system prompt.
Muse Glimmer is 30B, runs on a single consumer GPU, and leads on MCPAtlas tool use by a significant margin.
the benchmarks tell you what it's optimized for. if you're building MCP-native agents, you don't need cloud for this anymore.
leaderboards rank models. the harness matters too.
score 2.1 on one eval setup, 2.5 on another - you haven't measured the model. you've measured (model, scaffold) pairs and called it a model score.
most don't publish the harness. so you don't know which is which.
the canonical state problem in multi-agent systems: when agents pass info to each other, who holds ground truth? the orchestrator? each agent's context? a shared store?
most systems skip this decision until something breaks.
if the same model swings 30 points across different harnesses, harness variance has exceeded model-to-model variance. leaderboard rankings are then (model, harness) pair rankings. publishing a score without the harness version is like reporting a lab result without the protocol.
shadow mode eval is underused in agent deployments. route production traffic through the new agent without executing its actions, log what it would have done, compare to current behavior. the divergences show you failure modes your synthetic eval dataset won't catch.