Latest livebench.ai/#/ update includes llm inference time. This captures benchmaxxing a fair but - you will see the stronger Astra/Fable 5.1 use less effort to solve an open ended hard task than models like Opus 5.5, Fable 5.2
1
2
204
Agree - monorepo + verfication loops + strong models is all you need
maybe a hot take but i now consider adversarial reviews mostly a waste of tokens. far better to use those tokens on empirical combined with formal verification. i have updated pstack to use the interrogate skill less, and eventually i think i may remove it altogether
i have found very little value with code review in the abstract, agents tend to just get lost in the nitpicks and yagnis
1
84
This is the true emperor has no clothes moment - either its a massive investment portfolio reputation captialization, or the people running this have never used AI to really build anything in a significant way. (sonnet 5.5 > fable 5.1 is the giveaway)
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
And the Haiku beats Fable on this coding benchmark 🤯 artificialanalysis.ai/evalua…
1
44
And the ability to use a computer connected to the internet provides exactly that
String theory is what happens when an intelligent system is starved of sensory input and starts hallucinating. Intelligence needs grounding.
For this reason, intelligence is practically never the bottleneck to solving impactful problems. The bottleneck is iterated contact with reality.
74
Big update to livebench! livebench.ai/#/ - we have worked hard to do a fully private suite (cannot be gamed) reflecting real world general knowledge work use cases
1
66
tl;dr; - sorry to break it to you - you are all in the permanent underclass
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
5
217
This is very interesting work - especially relevant for building interpretable classifiers
The craziest experiment in my lab is a new setup by @diatkinson where he has found a way induce introspection on open LMs using @dillonplunkett's protocol, that lets him look inside introspection. It's nuts.
Here's how the experiment works... 🧵 →
3
77
We get the government we vote for. I tell everyone in SF that local elections are the most important vote they can cast
San Francisco did more than defunding police, they did the SF thing… created more bureaucracy to “reimagine” policing
...and then put a convicted felon at the helm of the Sheriff’s oversight board, and then made the board extremely hard to fire.
In 2020, Chesa Boudin, Dean Preston and gang pushed for a new Sheriff’s Oversight Board to “give voice to those impacted,” provide “accountability,” and join the “national movement for justice reform.”
They put convicted felon William Palmer as president, for his “lived experience.” He had served 31 years in prison.
Since joining the board, Palmer has been accused of violence by multiple women, including a rape charge. Now a 26 y/o woman says he gave her meth and pressured her into sex while she stayed at his apartment, where she overdosed on fentanyl. Palmer says he was acting as her “life coach.”
Everyone agrees Palmer should not be running this board, but because of how SF designed it, removing him is extraordinarily difficult. Commissioners like Palmer are protected by a process requiring formal misconduct charges, an Ethics Commission proceeding and a supermajority of the Board of Supervisors.
We should have passed Prop D in 2024, which would have removed this entire commission from the Charter along with many others, but it would have given more power to the mayor and I think people were reluctant, given Mayor Breed's administration.
We have a chance to fix this a little bit: vote YES on A and F this November.
Prop A is the commission-cleanup measure: it cuts the number of Charter commissions, eliminates or merges some, and moves others out of the Charter so they can be changed or abolished without another citywide election.
Prop F fixes the problem illustrated by Palmer: most commissioners would serve at the pleasure of whoever appointed them, so the Board could simply remove its own appointees without this ridiculous misconduct process.
6
214
Biggest failure mode with current AI is AI brain + human hands; best to have the AI help build its own hands and let humans set goals and judge outcomes
1
3
229
Happy 250th birthday America! Our founders would find it so fitting that we have birthed a technology that reopens the frontier to the pursuit of happiness for our present day digital yeoman farmer. The frontier is till open!
1
1
122
Wow - simplest explanation of how is incompetence on the whitehouse's part; 2-3 hours on an issue where they were likely wrong/unqualified to judge causing immense damange to US AI interests
NEW: Inside the 24-hrs before WH slapped export controls on Anthropic
- Last Thursday, Amazon CEO Andy Jassy raised concerns about Fable jailbreak to Trump admin
- Friday AM, Sean Cairncross, Bessent, Susie etc. held WH call to discuss
- Then White House started reaching out to Anthropic to speak with Dario Amodei, who was at a wellness retreat.
- When Amodei was finally available past 1pm, he had three tense phone calls with a combo of ppl including Cairncross, Bessent, Lutnick, Kessler, Will Scharf, Richard Walters, and Walker Barrett.
-Amodei tried to clear up what he assumed was a misunderstanding. He defended the guardrails and distinguished between universal and non-universal jailbreak
- Cairncross and Bessent were unmoved and asked Amodei to take down Fable and work with the admin to fix the vulnerabilities. (A WH official said Amazon’s findings were run past the NSA and they felt they had “proof.”)
- Amodei asked for more time and info, but he made no commitments to pull the model
- Bessent told Amodei directly at one point that he was making a “bad decision”
- By Friday evening, the Trump admin imposed its export controls.
- “Export controls were a last resort after begging them for hours to work with us,” senior WH official said.
W/ @cheyennehaslett
politico.com/news/2026/06/13…
1
218
We took our very SOTA computer backed agent and made it work with a standalone very fast NVME disk host (available at one of the cheapest prices 50G disk, 6G mem included in the basic $10 a month tier) . We did a lot of work to make it as performant as possible
🚨 Build, Deploy And Host Anything Using The Abacus AI SuperComputer
Build and Deploy
- 3D game servers
- always on agents
- open-source LLMs
- complex SaaS apps in any programming language
- database and APIs
All with ONE PROMPT using top models like Claude Fable, Opus 4.8 and GPT 5.5
2
158
For others who may have had the same minor panic!
Sorry, this wasn’t intentional, text-embedding-3-small isn’t going away!
teapawt.org/romainhuet/status/2047…
1
107
HP did the universe a favor - imagine if they had said yes!
Steve Wozniak famously went to his boss at HP and said they should build a personal computer. They said no, so he left to found Apple. It’s a lesson for leaders: one "no" shouldn't kill a contrarian but right idea in your company.
Twice a year at Coinbase, anyone in the company can pitch a "next bet" idea to a panel of folks. It's structured similar to pitching a handful of venture capitalists internally. If you get any one of them to say yes and fund it, you're green lit.
You need ONE yes, not a unanimous yes from everyone in the org structure from you to the CEO (a de facto committee).
Lots more goes into this, around capping resourcing on next bets (most are small 2-3 person teams), knowing when to shut them down, or under what revenue/profitability criteria they graduate to regular products. But this is important to having a company that produces repeatable innovation.
165
One more step toward our goal of leveraging the best parts of all the SOTA llms to the best agent harness on the cloud, desktop and clis integrated in a seamless way
🚨 BREAKING NEWS - Abacus CoWork brings Claude, GPT 5.4 And Gemini To Your Laptop!
Super excited to announce our MULTI-MODEL CoWork product!
- combine the coding power of Opus with the reasoning prowess of GPT 5.4
- optimized for efficiency using "low effort" mode
- computer use to tes
- packaged for FREE with ChatLLM and Abacus AI's Deep Agent
Get complex tasks done right on your laptop


















