How does @Shopify’s sidekick 🦸‍♂️ continue to get better, beat frontier models, and reduce serving costs by 96% Step 1: define what “good” looks like the most important step. rubric -> prod req -> scored criteria. these are passed to annotators who then turn conversations into ground truth. 2 best annotators annotate, compare w Cohen’s kappa, if 0.2 or below => rubric not clear, if higher => this is the ceiling. annotators need to include “why”. Step 2: calibrate your judges calibration takes the rubric (the first prompt) and turns it into something that can robustly evaluate thousands of conversations. they do this using DSPy, and calibrate w GEPA and others. this allows for offline evaluation, you can also backtest against previous A/B tests. keep each judge small and targeted. Step 3: beat the frontier baseline w autoresearch now that we have the judges we can optimize the entire frontier-powered system (but not the weight yet. they do this by configuring the entire system into a single markdown file: “where to get data, which directories the agent may edit, the judge as the metric, the optimizer to use, and the propose-evaluate-keep-or-discard loop.” then they use autoresearch to optimize everything from the prompts to the tool definitionss to the orchestration code - the entire harness. Step 4: weight updates once harness improvements plateau they set their sights on the weights. first: they looked for hard negatives in the prod data to expose where the model is weakest. “In a traditional workflow, each failure becomes a bug report or Slack thread. In the flywheel, these failures automatically enter a self-healing pipeline, where a panel of frontier reasoning models turns them into training signal that reinforcement learning folds back into the model’s weights.” then: panel of frontier models critiques each failure, an arbiter merges those in a repair instruction before the users turn (“hinting”). And the conversation is replayed from that point. If the judge passes it then it’s used a trajectory in reinforcement learning. If it still fails it’s passed to human annotators for the fix. training is in two stages. stage 1 - distill the healed trajectories into a smaller model using SFT. note: these are full trajectories including the reasoning that produced them. stage 2 - GRPO using the calibrated judges as the reward signal. For each prompt the model samples responses, the judges score them and GRPO reinforces those that perform best. “Supervised fine-tuning teaches the model successful trajectories to imitate; GRPO optimizes it directly against our definition of quality.” The self-healing pipeline runs daily. Training on both new and previous trajectories to limit drift and catastrophic forgetting across cycles. Step 5: prompt compression “A long prompt is a fixed tax on latency and serving cost, paid on every request.” Gist compression removes most of that tax. There’s a teacher with the full system prompt and a student with a short sequence of learned gist tokens. They train the gist token embeddings to match the teachers output distribution with the model weights frozen. The result is a small number of tokens that can reproduce the prompts behaviour at a fraction of the length and without loss in quality. THE RESULTS “It made the model better. The self-healing pipeline turns low-scoring production conversations into successful trajectories, giving the model a continual stream of lessons drawn from real merchant needs. Together, SFT and RL enable the specialized model to surpass the frontier model performance.” “It made the model far cheaper to serve. Serving this traffic on a frontier model could easily cost an estimated $27M per year based on average token costs. The fine-tuned model could come in at a fraction of the cost, closer to $1M: a 96% reduction in serving cost.” “The durable advantage is the loop that keeps turning production experience into better weights.” Full article 👇