sai (@skar312) header
sai (@skar312) profile picture

ai engineer in search of global maxima. bullish on agent traces & multiplayer ai.

Joined April 2025
Photo by sai (@skar312)Photo by sai (@skar312)Photo by sai (@skar312)Photo by sai (@skar312)Photo by sai (@skar312)Photo by sai (@skar312)Photo by sai (@skar312)Photo by sai (@skar312)Photo by sai (@skar312)Photo by sai (@skar312)
Pinned Tweet
My agent kept breaking in prod, and every fix I wrote broke something else. So I stopped writing the fixes. Now production failures become Harbor tasks, an RLM makes eval-driven reflections and GEPA optimizes the agent. Here's the whole flywheel:
7
3
1
8
1,834
weekend shenanigans with spinningup.openai.com/en/lat… (more people need to read this!)
12
My latest rabbit hole: getting more out of AI with less compute. MIT’s Efficient AI computing course by @songhan_mit goes from pruning and quantization to distillation, LLM deployment, and diffusion. hanlab.mit.edu/courses/2026-… Fall 2026 lectures are still dropping. This one’s worth bookmarking 📚
38
certified fresh.
I've started spending more of my time on evals than on the actual harness code. The agents get better by hill-climbing them. Every production failure becomes a test case. Every run gets traced. GEPA evolves the agent against the suite, and an RLM reads through the traces, including runs that passed in earlier generations, to figure out what to change. Sometimes that's a prompt edit. More often it's a new tool, a new skill, or a change to the harness. My job has narrowed down to two things: deciding what the agent should be good at, and keeping the evals honest about it. None of this works without some of the best open source tools around right now: @harborframework (@alexgshaw) for running evals in isolated containers at scale. @langfuse for traces you can actually query. @DSPyOSS for GEPA and RLM out of the box (@LakshyAAAgrawal and @a1zhang ) Each one is great on its own. Wired together, they're an agent that improves itself. Very few people are doing that yet.
1
54
I've started spending more of my time on evals than on the actual harness code. The agents get better by hill-climbing them. Every production failure becomes a test case. Every run gets traced. GEPA evolves the agent against the suite, and an RLM reads through the traces, including runs that passed in earlier generations, to figure out what to change. Sometimes that's a prompt edit. More often it's a new tool, a new skill, or a change to the harness. My job has narrowed down to two things: deciding what the agent should be good at, and keeping the evals honest about it. None of this works without some of the best open source tools around right now: @harborframework (@alexgshaw) for running evals in isolated containers at scale. @langfuse for traces you can actually query. @DSPyOSS for GEPA and RLM out of the box (@LakshyAAAgrawal and @a1zhang ) Each one is great on its own. Wired together, they're an agent that improves itself. Very few people are doing that yet.
My agent kept breaking in prod, and every fix I wrote broke something else. So I stopped writing the fixes. Now production failures become Harbor tasks, an RLM makes eval-driven reflections and GEPA optimizes the agent. Here's the whole flywheel:
1
1
5
307
sai retweeted
My agent kept breaking in prod, and every fix I wrote broke something else. So I stopped writing the fixes. Now production failures become Harbor tasks, an RLM makes eval-driven reflections and GEPA optimizes the agent. Here's the whole flywheel:
7
3
1
8
1,834
more people need to use harbor
32
small wins, 20 stars.
2
48
orchestratormaxxed with fable, ran out of opus.
1
60
the prodigal son returns.
42
Feed these rules to your Fable session and see the absolute symphony it orchestrates. - Route judgment to the Fable, execution one tier down to Opus or GPT-5.5, mechanical chores to Sonnet - Write the execution brief before the build. Freeze it. Unknowns raise as new issues, they don't get guessed mid execution. - Keep one reviewer blind to source. It attacks behavior on a real install. - Gate merges mechanically: re-run suites, tree-identity checks, fresh-install probes. - Assume agents die mid-task. Commit-as-you-go and resume-from-disk are the default, not the recovery. - Update the docs your agents boot from in the same iteration that changed the architecture. Stale instructions are inherited defects.
86
any consensus on what’s the best way for developing niche internal workflow agents? @LangChain deepagents or something else?
1
72
basically want something I can tune to do a verify specific task ( have a golden set) and a/b test across local and openai models. have used langgraph+dspy in the past but seems we have better options now
63
The moat in agents won't be the model. It'll be token cost. I ran one skill through an optimization loop and cut it 15× — 3,300 tokens a query down to 220 by grading every version on cost, not just quality. Every version scored an A. The grades were useless. Here's what the traces showed 👇 teapawt.org/skar312/status/2072019…
How to keep AI spend flat while token usage grows exponentially: Not with friction and spend alerts. With better defaults, routing, and caching. Better Defaults (not Usage Caps) – Engineers can choose any model they want, but defaults matter. We’re experimenting with defaulting to open weight models like GLM 5.2 and Kimi 2.7 through our LLM gateway, while still encouraging engineers to choose the right model for the task. 91% of our employees were never hitting their usage caps, so instead of lowering caps and driving up alerts, we're moving to cheaper defaults. Note that code reviews use a diversity of models, so they can check each other's work. Better Routing – In our custom harnesses, we preprocess prompts and route to the best model for the job, considering cache hits and model pricing. For instance, you may want a frontier model for planning, but not for execution where they can be overkill. Ultimately, humans shouldn't be choosing models - AI can automate this task. Better Caching – Cache misses are the easiest way to drive your cost up. All of our requests are cache aware, so we’re reusing a warm cache wherever possible. For example, our cache hit rate went from 5% → 60% in LibreChat once properly implemented. Keep Context Lean – Start fresh sessions when switching tasks. Scope file context narrowly. Disconnect unused tools. Don't just compact. The goal isn't fewer tokens used, it's fewer tokens wasted. Better Visibility – Our engineers can use as many tokens as they want, from whatever model they want, but we’ve made usage visible – and the more you spend on AI, the more impact we expect. The goal isn't to suppress usage. It's to build the infrastructure that makes exponential growth sustainable. Putting this into practice has cut our AI spend nearly in half, while our token usage continues to grow.
1
89
insane alpha on just mastering evals!
51
Marvel-based launch in THIS economy? Finally, someone who gets the need for a trust layer in AI.
Today, I'm thrilled to announce Pramaana's $27M seed, led by @khoslaventures. The foundational domains that hold the world together: tax, law, finance, healthcare; all run on certainty. Probabilistic AI can't give them that. We’ve been asked to accept wrong answers with AI as ‘hallucinations’, while in traditional software terms, it’s just a bug. And a wrong answer in such mission-critical domains is more than just a bug, it's a liability that could have catastrophic impact. We built Pramaana to deliver a 100% trustable experience to the domains that run on certainty: AI that is provably correct, not probabilistically correct. We turn statute and regulation into machine-verifiable code, so every output ships with mathematical proof of correctness. Our mission is to make AI take ownership of it’s work. Pramaana in Sanskrit stands for “means of valid knowledge”, and we’re going to achieve that by formalizing the world’s knowledge.
2
176
How are engineering teams handling agent + human collaboration at work? Not “which coding agent are you using?” I mean the messy org-level stuff: When every dev has their own agent, how do you stop 20 agents from independently rediscovering the same repo context? Are teams building shared skills for repeatable workflows? Shared memory for product decisions, infra quirks, debugging history, review patterns? Do agents write back what they learned, or does every session die in a chat window? How are people tracking token waste from agents asking the same questions over and over? Pair programming with agents is obvious. The harder question is whether agent work becomes shared organizational knowledge or just private autocomplete with a bigger bill.
69