@vikati

Benchmarking your favorite LLM

 San Francisco,CA
Joined February 2009
  • Tweets35
  • Following18
  • Followers299
3Media
Alex Vikati retweeted
@edwin 🌵📈
@edwinOct 2
1/7 We asked OpenAI’s Dots to build an AI status dashboard: one place to check @claudedevs, @cursor_ai, @OpenRouter, @GeminiApp, @MistralAI, @cohere API health. Twelve providers. Public page. Saved incident history. Private watchlists. Refresh every 15 minutes. We didn’t specify the stacks. We repeated this experiment 8x.
1
1
493
Resend continues its impressive run in our latest Fable benchmark. While custom/DIY rose to 21.4% overall and took ground in auth, caching, and feature flags, Resend defended the email category at 52%. It’s a reminder that when you make your product/docs the right agent building block, you stay picked even as models get dramatically better at building things themselves.
Zeno Rocha
@zenorochaJul 10
new Fable 5 benchmark is out! they ran Claude Code 810 times, with no tool names in the prompts. it would rather write code than install a dependency. it hand-rolled auth 64% of the time. 71% for feature flags. email? it picked @resend 52% of the time.
1
1
140
Full report: amplifying.ai/research/claud…

Amplifying | Coding agent intelligence

Benchmarking and Ground Control for developer tool providers. Understand agent choices and test what helps them succeed.

amplifying.ai
23
Alex Vikati retweeted
@edwin 🌵📈
@edwinJun 19
First raw volume. All-time, public GitHub: Claude Code: 12.5M commits Codex: 5.1M PRs Copilot: 3.8M commits Jules: 1.15M PRs Cursor: 1M commits Aider: 613k commits Devin: 161K PRs And those are floors. Private repos and squash merges never show up, so the real numbers are a lot bigger.
1
1
220
Alex Vikati retweeted
@edwin 🌵📈
@edwinJun 19
There’s almost no public data on how people actually use coding agents. So we’re building an index by tracking agent activity on public GitHub repos (commits, PRs, etc). Here’s what we found in our recent snapshot:
1
1
5
1,009
Alex Vikati retweeted
@edwin 🌵📈
@edwinApr 14
1/8 Mythos / Glasswing is clearly the main AI security story now: AI finding real vulnerabilities in existing production code. For most teams, though, this question is more immediate: Can an agent like Claude Code write a secure app in the first place?
1
4
1
3
950
The Claude Code leak shows a clear divide: select vendors gain small but compounding advantages. Everyone else gets generic UX + install friction. Direct impact on roadmaps and GTM.
@edwin 🌵📈
@edwinMar 31
1/9 The most interesting thing about the Claude Code leak for devtool companies: Anthropic hardcoded 120+ vendor names across 7 different systems in the source. Anthropic explicitly included your tool name in the code (or they didn’t 🤷🏻‍♂️) Thread 👇
1
1
148
Alex Vikati retweeted
@edwin 🌵📈
@edwinMar 23
After OpenAI/Astral acquisition announcement, we ran a benchmark on their tools. Turns out Astral tools were already a core part of the Codex (and Claude Code) workflow for Python developers. Ruff + uv came out on top in 75% of cases for linting, packaging, and pretty much everything else. Full report: amplifying.ai/research/astra…
1
1
233
Alex Vikati retweeted
@edwin 🌵📈
@edwinMar 20
1) Winners & losers from our OpenAI Codex vs Claude Code Picks benchmarks
1
2
1
2
381
Past category winners relied heavily on marketing and sales, and GTM still matters to get in front of coding agents. But agents won't keep using you unless you actually make them better (since the agents are trained with their harness). That shifts long-term value toward real product quality, not just distribution.
swyx
@swyxFeb 26
every category leader in this list should be worth at least $5b btw because koding agents will be recommending them for the next 5 years + infra is stickier than agents (full disclosure am smol resend angel)
1
1
85
Alex Vikati retweeted
@edwin 🌵📈
@edwinFeb 21
My friend @gentschev started using @Railway , @sentry and @resend for his homeschooltools.net side project because of Claude Code's recommendations. The impact is real.

Browse Offerings | Homeschool Tools

Browse curated homeschool curriculum, online schools, and educational resources. Find programs that fit your family's needs, budget, and learning style.

homeschooltools.net
Greg Gentschev
@gentschevFeb 21
Replying to @edwin
You’re right, Railway was a Claude Code recommendation. I’m also using Sentry and Resend. Your list seems pretty spot on.
1
3
375
Coding agents like Claude are a massive new distribution channel for infra providers. @Railway (~$124M raised) crushes peers like @render (~$258M raised) and @flydotio (~$115M raised) in Claude Code preferences. Will agent recommendations drive outsized growth?
@edwin 🌵📈
@edwinFeb 20
Replying to @edwin
3/9 Deployment (Python) Winner: @Railway – 82% picks. The dark horse: none of Vercel's hype machine, but quietly eating away tons of non-Next.js hosting from everyone (AWS, GCP, Heroku, Render, Fly, etc). When IPO? Simplicity wins. @JustJake Railway is stealthily dominating.
1
2
419
Winner and losers from our Claude Code Picks benchmarks!
@edwin 🌵📈
@edwinFeb 20
1/9 Winners & losers from our Claude Code Picks benchmarks that measure what Sonnet 4.5, Opus 4.5, and Opus 4.6 default to when building apps.
1
53
Alex Vikati retweeted
@edwin 🌵📈
@edwinFeb 18
Replying to @vikati
@vikati and I analyzed 2,430 Claude Code repo decisions. Claude Code never picked AWS or GCP for deployment. If agents are writing the first version of new projects, they’re influencing which tools get adopted at scale.
1
2
1
505
Alex Vikati retweeted
@edwin 🌵📈
@edwin13 May 2025
1/ AI is changing how brands appear in search. Try searching "best acoustic guitar under $500" in Google, ChatGPT, and Perplexity. You'll get three completely different sets of recommendations citing different sources
1
2
2
1,047
Alex Vikati retweeted
@edwin 🌵📈
@edwin23 Jan 2019
Speaking of AI bias & @AOC's comments, we did a bunch of tests and @azure, @awscloud, @clarifai & @IBMWatson all think @maddow is a man medium.com/@edwin/ai-thinks-…

AI Thinks Rachel Maddow Is A Man (and this is a problem for all of us)

In 2011, IBM Watson made headlines when it beat Jeopardy legends Ken Jennings and Brad Rutter in a $1M match. In Final Jeopardy, Jennings…

medium.com
2
3
2
8
We just released a new Tether dataset that covers every transaction up to block 546906. Go to blockspur.com/tether/downloa… if you are interested in doing primary Tether data analysis.
Alex Vikati
@vikati19 Oct 2018
I just published Tether vs Other Stablecoins medium.com/@vikati/tether-vs…

Tether vs Other Stablecoins

A data-driven comparison of Tether, Circle USDC, Gemini Dollar, Maker Dai, Paxos Standard & TrueUSD

medium.com
1
1
9
I just published Tether vs Other Stablecoins medium.com/@vikati/tether-vs…

Tether vs Other Stablecoins

A data-driven comparison of Tether, Circle USDC, Gemini Dollar, Maker Dai, Paxos Standard & TrueUSD

medium.com
2
8
4
23
I just published medium.com/@vikati/ranking-e… - a data-driven post on the actual usage of Ethereum smart contracts. In the long run, it is users, traffic, and revenues that will drive real value for decentralized networks.

Ranking Ethereum Smart Contracts

A data-driven review of the actual usage of smart contracts: “traffic,” “users,” & “revenue”

medium.com
1
3
9
I just published “How Private Are Privacy Coins: A Closer Look at Zcash and Zclassic’s Blockchains” medium.com/@vikati/how-priva…

How Private Are Privacy Coins: A Closer Look at Zcash and Zclassic’s Blockchains

A Data-Driven Review of Zcash & Zclassic’s Network Privacy

medium.com
2
18
4
68
Load more