Mohamed Nanabhay

🔀 Dynamic Model Escalation For Efficient Evals 💰 Most eval pipelines pick one judge model and run it on everything. If it’s a big model, it’s too slow and too expensive to score production traffic, so teams sample a few percent and hope. If it’s a small model, it’s cheap enough to score every trace, but nobody knows which of its scores to trust. The question isn’t which judge model to use. It’s when the judge should admit it doesn’t know. Here is what we have built at @fiddler_ai as part of our Centor Evaluation Runtime. Two models function with two distinct jobs. A decisioning model sits in the hot path and scores every span/trace in under 100ms. A reasoning model sits behind it and only ever sees what the decisioning model sends it. The interesting part is the handoff. The decisioning model doesn’t just emit a verdict. It emits a verdict plus an uncertainty score about how much that verdict should be trusted, derived from the trace itself and from where this verdict sits relative to everything the model has scored before. That signal is what routes. Traces the decisioning model is sure about keep their score, and that’s the majority of traffic. Traces where the signal says the model is operating outside its competence go to the reasoning model, which re-scores them with the time and context a fast model can’t afford. So the split isn’t fast versus accurate. It acutally works as decide JIT versus deliberate post-hoc analysis. The decisioning model’s job is to be right quickly and to know when it isn’t. The reasoning model’s job is to be right on the cases that need thought. One eval score comes out the other end, and the user never needs to know which model produced it. Ofcourse we keep an audit of all of this if user wants to actually know the details. This is very similar to how humans perform a good on-call rotation. The pager doesn’t wake the senior most engineer (large reasoning model) for every alert. It wakes them for the ones that the on-call engineer (fast decision model) using the the runbook can’t resolve. So, what are the main advantages of this: The reasoning model’s bill is a function of how unusual your traffic is, not how much of it there is, which is the only way scoring every production trace stays affordable. And the escalation queue becomes the most useful dataset you have: a running list of where the decisioning model is weak and where your traffic is drifting. When you label those, feed them back into the next version of the decisioning model, and the escalation rate drops over time. The eval system tunes itself on its own misses. Welcome to the world of self-tuning AI Observability! Eval Models: fiddler.ai/centor-models