Open source agentic coding harness where local and frontier models live in harmony.
Joined January 2009
- Tweets4.5K
- Following340
- Followers4K
- Likes7.3K
Albatross retweeted
I am going to start to share the leaderboard at VulcanBench more regularly.
VulcanBench is the only benchmark I know that benchmarks across effort levels, on an eval suite of 100% real engineering tasks.
Getting ready to sunset eval suite 3 and eval suite 4 is almost ready to go.
Here's the current leaderboard for Eval Suite 3, Grok 4.5 High is the highest scoring model still with Fable 5 Low, yes Low, right behind it.
What I've been able to uncover with VulcanBench is that way too many people are running models at Max effort, thinking that buys them more accuracy, when really it just costs more and uses more tokens so takes more time.
If you're still in the mode of, tell my agent to do something then go get coffee, you're probably still living in the past. You can move faster, with higher accuracy, the key is not thinking you need Max effort all the time.
Time to share the leaderboard for VulcanBench, Eval Suite 3.
As a reminder, these are 100% real engineering tasks, from real repos. No puzzles, no math problems, no exotic architecture.
Our leaderboard takes the data we have from our full effort sweeps, every effort level, and reports the effort level that gave the highest accuracy.
Right now, so many engineering teams just run a model on High or Max, assuming that's what they need to get the best accuracy.
But for most regular, routine, engineering tasks, the stuff we do day in and day out, that's overkill. And it means the company you work for is paying a LOT more money for tokens, all to get the same accuracy, just a lot slower.
Currently Grok 4.5 High still tops the chart at 90, with Fable 5 Low right behind it at 89 and DeepSeek V4-Flash Max right behind that at 88.
If you're using Grok 4.6, Medium is actually the sweet spot for effort, and Opus 5, which everyone seems to be complaining about these days...well you're probably using it on High, which actually doesn't perform as well as low.
Effort matters.
Live long and benchmark 🖖
Albatross retweeted
Factor investing is what made me stand out at JPMorgan.
But it took me years to master the information coefficient.
In 1 minute, I'll teach you the 10 things you need to know (that took me 1 year to learn).
Let's go:
Albatross retweeted
If you think the YC deadline has “passed,” you might be walking away from your shot.
YC still reviews late apps if you fit what they’re looking for this batch.
You can literally:
– Write your app in a day
– Submit
– Hear back in 24–48 hours (if you're a fit)
– Be on a flight to SF for your interview
Don’t self-reject. ☝️
Albatross retweeted
Our latest newsletter is out!
- Welcome to the Video-First Economy
- The 10 Essentials to Building a Values-Driven Company
- Mastering the "Frugalnomics" Sales Environment
- A Year of Firsts
- Company news, open roles & more
mailchi.mp/emcap/august20
Albatross retweeted
Congrats to @NGhamsari & the @blendlabsinc team! We're incredibly proud of all your hard work and dedication. 🦄✨
cnbc.com/2020/08/12/digital-…
Albatross retweeted
Growing up with the Atari 2600, my first gaming crush mashable.com/article/atari-2…
Albatross retweeted
Tutorial: Tracking Verisign’s Monthly Top Popular Keywords using DomainScope API, PHP, and MySQL - youtube.com/watch?v=iNM5FDFN…
Albatross retweeted
In South Korea 🇰🇷, the solar panels in the middle of the highway have a bicycle path underneath - cyclists are protected from the sun, isolated from traffic, and the country can produce clean energy. teapawt.org/UniverCurious/status/1…
Albatross retweeted
When we look back in five years, we are going to realize that there were 10 to 20 amazing companies that were started that changed the world and led us to a brighter future. Ask yourself: “Why Not Me” or “Why Not Us”. Now is your time. The world is waiting.
Albatross retweeted
BREAKING: NYC restaurants can only serve takeout and delivery options starting Tuesday, Mayor Bill de Blasio orders. Bars that do not serve food must close.
Albatross retweeted
Having an extremely hard time not becoming livid at friends who are not practicing social distancing. Even if you are fine, YOU CAN KILL PEOPLE. How irresponsible and selfish must you be not to grasp this concept?!
Albatross retweeted
Want to optimize your For Sale Landing Pages? Read what @doronvermaat said about those, and more: jasonofflorida.com/efty-land…
ps - Thank you Doron for a candid and awesome interview!
#DomainInvesting #DomainNames
Albatross retweeted
If this was you last night: Please reconsider tonight.
Somewhere there is a little girl with leukemia, a husband with HIV, a grandmother with lung problems. Maybe you think you're invincible, but they aren't.
You’re unable to view this Post because this account owner limits who can view their Posts. Learn more
Albatross retweeted
I landed in Haiti and as I got off the plane I was tested for #COVID19 let that sink in
Albatross retweeted
Lots of people have been asking how we acquired hey.com, so I wrote up the story. Actual emails included. Here’s how we acquired hey.com —> m.signalvnoise.com/how-we-ac…