Haritha Nair (@margharithaa) header
Haritha Nair (@margharithaa) profile picture

Building @oqoqo_ai - the easiest way to build evals and custom benchmarks at frontier scale. sign up at https://teapawt.org/t.co/akEJoFCmML, prev. @windsurf @microsoft

San Francisco, CA
Joined November 2021
Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)Photo by Haritha Nair (@margharithaa)
Super relevant info from @vercel but need to keep in mind - evals often hide real issues agents face when using your MCP and other tools because it might result in an eventual success - it's a lot of infra to manage if you run them at scale - repeatable evals are critical to monitoe actual improvements over time We've solved for all at @oqoqo_ai but always interesting to see how leading companies are approaching it
1
2
99
Haritha Nair retweeted
so cool to present a deep dive on 'evals and how agents choose or ignore your products' with @apify team. that too right after their flagship conference 🚀
Replying to @0xLukasB
Shoutout to everyone who demoed what they've been cooking: @freestyle_dev @xulux_ai @oqoqo_ai @braintrust @mcpjams Couldn't have asked for a better send off to conclude my SF trip 🙏🏻
1
3
41
At #RunByApify and I wish more tech conferences would present ways people are using their products for social good. Seeing canary.ngo present shouldn't have been as surprising as it was @apify
1
9
379
Haritha Nair retweeted
So bullish on MCP. Its utility exceeds the "M" part: even AI aside, it'll be a civilization-scale accomplishment to get all our systems communicating through one universal open protocol. A missing piece was how to get arbitrary callbacks across sessions. MCP could read and write, but it couldn't tell you later "this event happened." MCP Events fixes this. We're adding support in our own MCP. Say an agent makes a deployment, and wants to know when it's complete. Instead of polling, it can subscribe to 𝚍𝚎𝚙𝚕𝚘𝚢𝚖𝚎𝚗𝚝.𝚜𝚞𝚌𝚌𝚎𝚎𝚍𝚎𝚍. PSA: With Vercel Queues and Workflows, writing the backend machinery for your own MCPs to do this is easy peasy.
Vercel MCP now publishes events. Agents can subscribe to events like 𝚍𝚎𝚙𝚕𝚘𝚢𝚖𝚎𝚗𝚝.𝚜𝚞𝚌𝚌𝚎𝚎𝚍𝚎𝚍 and 𝚍𝚎𝚙𝚕𝚘𝚢𝚖𝚎𝚗𝚝.𝚎𝚛𝚛𝚘𝚛 and get webhook notifications when they fire. No more polling for changes. Changelog → vercel.com/changelog/vercel-…
169
64
20
1,073
89,635
Haritha Nair retweeted
LLMs are still SO bad at building tools for agents. Every time I try to make Opus set up an MCP authentication path, it makes horrible assumptions and builds something that straight up doesn't work
253
38
43
2,151
169,853
Who knew being a founder is mostly about messaging people and having an identity crisis while waiting for people to reply
1
50
Haritha Nair retweeted
Five years ago @adamvpro and I started @zenml_io because getting AI from a notebook into production was painful. It still is. Only now it's agents, not just ML models, that make it hard. Building an agent takes an afternoon. Keeping it working is the hard part. A prompt tweak breaks something that worked yesterday. A cheaper model quietly stops answering. You only hear about it when customers complain. Every team we talked to said the same thing: we don't need another dashboard, we need someone to do the work. So we built that. ZenML Pro now comes with an always-on AI engineering team. It reads your traces, finds what your evals miss, and proves the fix works before you ship it. This tech is backed by ZenML and Kitaru, our two OSS products, that stay open source and freely available. To kick off this new chapter, we have decided to rebrand. ZenML is now ZenML Labs, and our ambition is greater than ever. Connect your agent traces today, free at zenml.io. We are very proud of the onboarding ;-) We wrote about it here: zenml.io/blog/introducing-ze…
18
13
5
69
16,612
Haritha Nair retweeted
people are like “why do LLMs always think they’re in an eval lol” and then have bad dreams about forgetting to turn in a homework assignment 10 years ago
78
351
16
6,422
135,397
Haritha Nair retweeted
An open secret among fast-growing startups: 𝗻𝗮𝗶𝗹 𝘆𝗼𝘂𝗿 𝗮𝗴𝗲𝗻𝘁 𝗲𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲. Here's how to get it right: 𝟭/ 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝗱𝗶𝘀𝗰𝗼𝘃𝗲𝗿𝗮𝗯𝗶𝗹𝗶𝘁𝘆. Identify 20 representative prompts your buyers use. Have an agent run them weekly in fresh sessions across relevant models, track mentions, and inspect sources behind them. Let it improve your content and measure the results. (1/10)
1
1
2
42
Randomly remembered #include<iostream.h> Does it remember us too?
2
39
Running your 1st eval is easier than you think been testing agents on real knowledge-work tasks to figure out what I can trust them with. I expected to spend time comparing answers. I ended up questioning the exams, the instructions, & the grading
12
1
13
811
I interned for a summer in AWS. Now everyone I knew at AWS is at Microsoft and vice versa 🤷
45
Agentbox 2.0 🎉🎉
Replying to @gmi_cloud
@oqoqo_ai runs agent evals on Agentbox. One experiment: 300 tasks, 8 agents, 5 models. That's 12,000 runs, each in a fresh sandbox. They submit the full grid at once, with up to 50,000 concurrent sandboxes per experiment and no sandbox infra to maintain. gmicloud.ai/blog/thousands-o…
47
Haritha Nair retweeted
Evals are the most underrated skill in AI engineering. 1. Start with error analysis, not metrics Sample 100+ production traces. Annotate each failure in free text (open coding), then cluster them into failure categories (axial coding). Count frequency per category. Your top 3 categories are your first metrics. 2. Build a versioned golden dataset Store inputs, expected outputs or rubrics, and metadata (intent, difficulty, user segment) in a versioned dataset. Stratify it so rare but critical cases aren't drowned out by easy ones. Version it alongside your prompts so every score is reproducible. 3. Generate synthetic data along explicit dimensions Don't just ask an LLM for "test questions." Define dimensions (persona × intent × complexity × edge case), generate combinations as tuples, then turn each tuple into a natural input. You get coverage instead of 200 near-identical examples. 4. Deterministic assertions first JSON schema validation, regex checks, required fields, length limits, code execution, SQL that actually runs against a test DB. These are cheap, fast, and have zero variance. Reserve model-based grading for what code genuinely can't check. 5. Evaluate retrieval separately from generation For RAG, measure retrieval with recall@k, MRR, or nDCG against labeled relevant chunks. Measure generation with faithfulness (is every claim supported by retrieved context?). If you only score the final answer, you can't tell which half is broken. 6. Trajectory evals for agents Log every step: tool name, arguments, result, tokens. Score tool-call precision and recall against a reference path, argument correctness, step count, and error recovery. An agent that reaches the right answer in 14 calls when 3 would do is a cost and latency bug. 7. pass@k vs pass^k pass@k = succeeded at least once in k runs. pass^k = succeeded in all k runs. At 80% per-run success, pass^5 drops to roughly 33%. Production reliability lives in pass^k, so run each case multiple times at your real temperature. 8. Simulated users for multi-turn Give an LLM a persona, a hidden goal, and constraints, then let it converse with your system until the goal is met or a turn limit hits. Grade goal completion, turns-to-resolution, and policy violations. Single-turn evals miss most state and memory bugs. 9. Align your LLM judge like a classifier Hand-label 100+ examples as pass/fail. Split into train/dev/test. Iterate on the judge prompt using dev, then report TPR and TNR on test. Use binary outputs with a written rubric, require a short critique before the verdict, and pin the judge model version. 10. Control judge bias in pairwise comparisons LLM judges have position bias and often prefer longer answers. Run every A/B comparison twice with swapped order and only count consistent verdicts. Normalize or cap length if verbosity is skewing results. 11. Respect statistics With 100 examples at 80% accuracy, your 95% confidence interval is about ±8 points. A jump from 80% to 83% is probably noise. Use bootstrapped confidence intervals, or a paired test on the same examples, before declaring a win. 12. Gate deploys in CI Run the eval suite on every PR that touches prompts, models, tools, or retrieval config. Set per-category thresholds instead of one global score, so a big win in one area can't hide a regression in another. Fail the build on a regression. 13. Online evals in production Sample a percentage of live traffic, run your judges asynchronously, and alert on drift per failure category. Use shadow deployments to run a new prompt or model on real traffic without showing users the output. 14. Make cost and latency first-class metrics Track tokens per request, cost per successful task (not per call), and p50/p95 latency next to quality. A change that keeps quality flat while cutting cost 40% is a real win and deserves to ship. 15. Close the loop Every production failure becomes a new labeled test case. Every new model release gets run against the full suite before anyone debates switching. Over time your eval set becomes the most valuable asset in the codebase. Prompts are disposable. Models get replaced every few months. Your eval infrastructure is what compounds.
129
149
37
2,193
240,440
Did 1000+ runs on @oqoqo_ai to see the most common failure patterns in skills. Codex, Claude code, opencode runs done on devtools including @posthog @resend @p0 @ExaAILabs @WorkOS @firecrawl @linear
3
4
362
How are people releasing their own decisions models so quickly? Did all of them have this in their pipeline and jev just happened to release first? What am I missing?
60
Wondering if I can somehow make friends with all the people who showed up at @mercury bookstore
1
7
224
SF tech week 😭
5
17
419
SSI was ahead of its time
47
Haritha Nair retweeted
probably the best tool on the market for finding any company’s full tech stack runsonwhat.com
1
2
119