Make AI agents production-ready through independent evaluation and training.
Joined January 2024
- Tweets119
- Following15
- Followers184
- Likes204
Pinned Tweet
Recently we built OTelBench – a benchmark to test how well LLMs handle OpenTelemetry instrumentation.
We tested 14 models. The best (Claude Opus 4.5) hit only 29%.
These weren't trick questions, just small subset of typical SRE tasks.
Link here:
quesma.com/blog/introducing-…
Quesma retweeted
We brought @QuesmaOrg software to Silicon Valley on floppy disks; each could hold 370k tokens. Very positive reactions at AGNTCon 2025.
Quesma retweeted
How to maximize the value of agentic coding? Use insights from session transcripts.
@QuesmaOrg Booth at AGNTCon + MCPCon North America 2026 in San Jose. Back left corner.
See our demo at the booth in the back left corner, or at quesma. com.
Quesma retweeted
For the first time, we got over 2048 visitors in the last 30 minutes on the @QuesmaOrg blog, thanks to a Hacker News home run by @pmigdal . You should read it; it's beautiful. 🧵
Quesma retweeted
This morning, Claude Code burnt 113M tokens with 202 subagents for my research (Opus 5.5). It would cost $72 at list price, though I ran it on a Max x20 subscription.
I simulated optimizations with @QuesmaOrg : grouping agents would make it worse (up to +83%). Leaner prompts, smaller tool outputs, and fewer redundant checks would cut it 30-50% on the same model. Even more with a cheaper one.
Quesma retweeted
In 2011, shipping a demo overnight was impressive.
In 2026, vibe coding makes that cheap.
So we gave students 18 GB of coding-agent transcripts and one Friday evening to find where we burn tokens.
@QuesmaOrg First token economics hackathon:
quesma.com/blog/our-first-ha…
Quesma retweeted
GPT-6 Astra rocks at puzzles - 3d game Portal, Baba Is You, ARC-AGI-3, MazeBench.
Is there a puzzle game still to hard to for this model?
Or maybe all that remains is unsolved historical codes?
quesma.com/blog/gpt-6-astra-…
Quesma retweeted
Are terminal compression tools actually saving us money?
Together with @jakozaur we tested one of the most popular ones across 1,740 Terminal-Bench 2.1 runs.
Huge reported token savings.
Almost no improvement in the final bill.
Token compression ≠ cost optimization.
quesma.com/blog/does-rtk-mak…
Quesma retweeted
Quesma benchmarked RTK (Rust Token Killer), and our post landed on the Hacker News homepage. Up to second place:
news.ycombinator.com/item?id…
Quesma retweeted
With new models and record-high results on benchmarks, do LLMs know it all?
Not really, if you dare to go outside established norms. As @pmigdal showed in the latest @QuesmaOrg post on Hacker News. The LLM still could not reliably tell which mushrooms are edible vs. fatal.
Quesma retweeted
Wrote a new blog post for @QuesmaOrg :
quesma.com/blog/codex-pricin…
Thanks, @pmigdal and @bartkotrys, for your feedback.
Quesma retweeted
gemini 3.6 flash cleared 88% of the baba is you intro levels and spent $124 doing it
gemini 3.7 flash cleared 100% and spent only $5.38
and boy, is she fast! 😍⚡️ @QuesmaOrg
quesma.com/blog/baba-is-aug-…
Quesma retweeted
Quantization is better than a lot of ppl think, see @pmigdal at @QuesmaOrg blogs such as:
quesma.com/blog/qwen38-27b-q…
Quesma retweeted
Same tokens, same model, up to a 40x price gap. Priced my own Claude Code usage at API list rates:
- Max 5x ($100/mo): $2,092 (21x)
- Max 20x ($200/mo): $2,986 (15x)
The seat plans are a subsidy. Enjoy the buffet while it's open.
Full math: quesma.com/blog/claude-code-…
Quesma retweeted
Some said if you don't like Fable pricing, you can use subagents with cheaper models. At @QuesmaOrg @bartkotrys tried it and benchmarked against Terminal-Bench, and it backfired in four ways:
quesma.com/blog/tbench-orche…
Quesma retweeted
Can LLM quantizations harm the pelican?
I tested all @UnslothAI's quantizations of Qwen 3.6 27B to see if they can draw @simonw's SVG pelican riding a bike.
quesma.com/blog/qwen-quantiz…
Quesma retweeted
Things I shipped last weekend:
1. Offsite @QuesmaOrg presentations and photos, hosted on our own site behind SSO. Live the day after the event.
2. Animated visualizations for FlexDrill, because the founder asked.
🧵 1/4
Quesma retweeted
Yesterday, Frontier-Bench landed 0.1 with tasks from @pmigdal and @QuesmaOrg . Today, Opus 5 was released, and it was put in the spotlight.
Quesma retweeted
AI + Ghidra by NSA = reverse-engineering fun
I am speaking at @AITinkerers Warsaw, 4th Mar 2026.
One of my favorite event series - by and for the creators community.
Vibe-resurrecting an old game from binaries 👾 and vibe-hardware-ing a LED backpack 🎒🌈.
