Our mission is to raise AGI with the richness of humanity — curious, witty, imaginative, and full of breathtaking brilliance.
Joined June 2020
- Tweets769
- Following144
- Followers9.4K
- Likes274
Surge AI retweeted
one of my fav findings from sudo L7:
opus 5.5 and gpt-6.1 sol are basically tied at writing correct code. but opus is much better at the behaviors that get you promoted to staff.
on functional correctness:
opus 82.1% ~ sol 82.4%
on thought partnership (e.g., telling you when there’s a better approach than the one you asked it to build):
opus 76.1% >> sol 53.0%
on communication (explaining its decisions and the tradeoffs it makes, plus any limitations):
opus 66.4% >> sol 49.4%
put differently: imagine promoting an engineer because their unit tests pass. that's basically how we've been evaluating coding agents.
but staff engineers do a lot more than that. they also:
- question bad assumptions
- suggest better approaches
- think about tradeoffs
- and help everyone around them make better decisions.
that’s what we want the best agents to do too, and that's what we built sudo L7 to measure.
Surge AI retweeted
another Surge benchmark on the Muse Wonder model card 👀
Muse scores 33.2% on GDP.pdf. #3 on our leaderboard, right behind Astra
nice work @Meta!
Surge AI retweeted
Sweet new benchmark from Surge looking at how close AI is getting to a staff-level software engineering. We're half-way there.
introducing sudo L7, our new benchmark for staff-level software engineering.
coding agents are getting very good at being L3s. give them a well-defined ticket and they can write excellent code.
but a staff engineer is valuable for everything that happens around the code.
what should we build? what already exists? what's going to break in prod? what did the tests miss? should we even be doing this?
sudo L7 has 60 tasks, mostly in private production repos from real companies, written by engineers who've actually owned these systems.
expert rubrics grade everything from functional correctness to engineering craft, architectural judgment, thought partnership, and unnecessary complexity.
the best agents succeed on only ~45% of tasks.
coding is only part of the job. for now, we're keeping them at idiot genius L3.
Surge AI retweeted
introducing sudo L7, our new benchmark for staff-level software engineering.
coding agents are getting very good at being L3s. give them a well-defined ticket and they can write excellent code.
but a staff engineer is valuable for everything that happens around the code.
what should we build? what already exists? what's going to break in prod? what did the tests miss? should we even be doing this?
sudo L7 has 60 tasks, mostly in private production repos from real companies, written by engineers who've actually owned these systems.
expert rubrics grade everything from functional correctness to engineering craft, architectural judgment, thought partnership, and unnecessary complexity.
the best agents succeed on only ~45% of tasks.
coding is only part of the job. for now, we're keeping them at idiot genius L3.
Surge AI retweeted
🥖 LGTM, le Chonk 🥖
in their latest Mistral Large 4 release, @MistralAI ran a blind Surge human eval across five frontier models - professional software engineers comparing five frontier models on coding quality. the big guy finished #2, first among open-weight models and behind only Opus 5.
most benchmarks can narrowly measure correctness. but human evals get at the deeper stuff: the professional taste, judgment, and maintainability we look for in the real world as well
the difference between passing a set of unit tests vs. getting a "LGTM - ship it"
bien joué, le Chonk ! congrats to the entire Mistral team 🇫🇷
Meet Mistral Large 4, aka Le Chonk.
• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.
Open weights release end of October.
Surge AI retweeted
A big focus for our team at @Pulse__AI is making the data AI agents receive more useful.
Huge congrats to the @HelloSurgeAI team on GDP.xlsx, a strong benchmark testing spreadsheet reasoning across finance, engineering and healthcare.
With Ultra 2, our latest model, we’re focused on preserving hidden content, formulas, complete records and visual context within these enterprise documents. We evaluated Pulse’s spreadsheet extraction pipeline using the benchmark’s OpenHands agent harness, providing structured text and image evidence upfront.
Our latest reported task-pass rates, compared with our earlier Pulse-assisted results:
- Claude Opus 5.5 46.4% (up from 41.4%; +5 percentage points)
- GPT-6.1 Sol 37.9% (up from 32.9%; +5 percentage points)
- GPT-6 Astra 35.0% (up from 30.0%; +5 percentage points)
- Claude Fable 5.1 33.6% (up from 28.6%; +5 percentage points)
- Gemini 3.8 Flash 23.6% (up from 18.6%; +5 percentage points)
Better data gives agents more to work with, read more here: runpulse.com/blog/pulse-gdp-…
Surge AI retweeted
GDP.pdf just got a little brother: introducing GDP.xlsx
the world runs on spreadsheets. somewhere in Sheet2!D18 there’s a vlookup that changes a $14m decision.
GDP.xlsx has 70 real-world tasks across 12 domains. what’s hard is figuring out how the workbook even works:
→ which tab is stale?
→ what does a red cell mean?
→ which comment explains the exceptions?
→ which chart does Finance actually trust?
→ did you remember to check the termination clauses in Sheet 3?
the best frontier agent scores 38.3%.
(congrats Gemini 4 Argon!)
this one’s for you, Kelly.
Surge AI retweeted
New research:
What Does GDP.pdf Teach, Beyond PDFs?
GDP.pdf has become a widely used frontier eval, so we wanted to know: if we train on the underlying skill, do we just get better at PDFs? Or do we learn something more fundamental?
So we post-trained Kimi K2.7 on 1,465 GDP.pdf companion tasks.
As expected, GDP.pdf improved substantially:
11.0% → 24.5%
But GDPval improved too: full-task success more than doubled, and 90% rubric pass rate jumped +13.2pp - including on tasks with no PDFs, in an environment with files, tools, spreadsheets, and finished deliverables.
Why? What behavior transferred?
Before training, Kimi often grabbed the first plausible source and started working.
After training, it looked around first: what's here, what matters, is anything missing?
On one task, the base model opened the first weekly timesheet and prematurely started calculating a monthly report. Incorrect!
The trained model correctly found all four weeks first, combined them, and then started working.
My favorite way to think about this:
A PDF is a small evidence environment. A good professional PDF forces you to figure out which section a fact lives in, which charts matter, how the pieces fit together, and whether you have enough evidence from Sections 1 and 2 to act.
Based on trajectory analysis, our hypothesis is that training on rich PDFs taught the model to gather all the evidence before it started acting.
After 1,465 tasks, Kimi kept that habit even when the task moved beyond PDFs.
Surge AI retweeted
can agents survive a 9 to 5? introducing DAYJOB, our new benchmark for economically valuable agents.
GDPval asks: “Please reformat this document. Here are 337 words explaining where to look, what analysis to run, and exactly what the finished thing should look like.”
real life: “@bob is that ready yet"
("that" = a forecast spread across 10 spreadsheets, 8 docs, a slack thread where finance is fighting with ds, and a file called model_FINAL_v3_DONT_USE_THIS.xlsx)
DAYJOB is our new benchmark family for knowledge work agents: realistic requests, messy environments, and end-to-end professional judgment.
instead of being told exactly how to do the task, we tell the model what Bob gets told - and reasoning through the rest is the test.
~~
after all, models can do extraordinary things when the task is clear.
summarize a document. clean up an email.
solve navier-stokes!
but somehow we have not yet automated Bob.
the real world is rarely clear. steps are easy; what's hard is the messy end-to-end judgment around them.
real work doesn't arrive with step-by-step instructions on what to do. you get a short vague request. then you have to:
→ figure out what "that" means
→ find which files matter
→ know when to ignore what sally in finance wrote in slack
→ decide which assumptions are outdated
→ notice on day 3 that something from day 1 was wrong
GDPval was a big deal. it was an important step toward measuring real professional work.
but its prompts average 337 words and mostly tell the model how to do the job. the typical task comes with one file. in openai's own experiments, when they shortened the prompts, scores dropped because models struggled to "figure out context."
figuring out context is most of bob's day.
so we built DAYJOB. prompts are short (5x shorter!) and realistic. it comes in much larger environments, averaging 20 to 26 files, and take 20+ hours of expert work. the model has to figure out the job, not just execute a checklist.
~~
today, we’re launching with DAYJOB: Healthcare + DAYJOB: Finance, with more professions coming soon.
one task: review a fund's reports before they go to a new client and flag anything that should stop them.
buried in the pack is a position priced off a bloomberg screenshot. the price was in south african cents. someone entered it as rand.
$260k became $26m.
we collected 66 trajectories across 22 model configs. only 2 caught it.
congrats opus.
everyone else did a lot of impressive work around a $26m mistake.
~~
so can agents survive a 9 to 5?
the best model passes 24.7% of healthcare assignments and 23.9% of finance.
bob's job is harder than navier-stokes.
Surge AI retweeted
one of my favorite examples from our Kimi coding post-training run:
it had to write a Zstandard decompressor, but there was no zstd binary available to check whether it worked.
so the trained model wrote a compressor first, generated its own valid test files, and used those to test the decompressor.
no ground truth existed, so it built one itself.
Read the full report here:
surgehq.ai/blog/hill-climbin…
Surge AI retweeted
i agree with dario.
i used to work on feed algorithms at google, fb, and twitter. we optimized for engagement because it was the only thing we knew how to measure, even when it was only a weak proxy for what people actually wanted.
and then the systems did exactly what we asked! they figured out how to hack engagement in ways nobody intended, and we didn't understand the consequences until later.
for me, that's always been the alignment problem in miniature. that experience is a big part of why i started Surge, and it shaped how i think about data, quality, evals, and the hills we want to climb.
AI makes the same problem even more consequential. we have models that can solve Navier Stokes, that billions of people will talk to and trust, and that can act on their own.
getting them to climb a hill isn't the hart part; they're already incredible at that. the hard part is choosing the right objective - and making sure the data, rewards, and evals actually point at what we want.
so, i agree. if we don't want to repeat the lessons of social media - but with way smarter systems that can act on their own, and convince, at way greater scale - alignment has to be built into the way we train and evaluate models from the beginning.
a lot more work is needed to make sure models stay human-aligned. reach out if you're interested in working on this together.
Surge AI retweeted
An intern knows how to write code. A staff engineer knows how to make it *shippable*.
We post-trained kimi k2.7 and that’s basically what changed.
Before RLing, kimi k2.7 was failing in the last mile: dropping requirements, writing narrow tests, creating regressions
After RL on just 1,700 Surge coding tasks:
+20.0 SWE-Marathon
+14.6 Terminal-Bench 2.1
+12.4 DeepSWE
+10.7 Terminal-Bench 3
+4.7 SWE-Bench Pro
Surge AI retweeted
@ArtificialAnlys just added GDP.pdf to their Intelligence Index.
Which means their definition of intelligence now includes something deceptively simple: can models understand the documents you deal with on a normal Tuesday at work?
Leases, invoices, dosage tables, and financial reports.
Astra, the best model, still solves just under 1 in 3.
Master the boring, master the frontier.
Announcing Artificial Analysis Intelligence Index v4.2. We are accelerating elements of our upcoming v5 release with interim updates to keep pace with the frontier. Index v4.2 has more complex and realistic tasks, and more private test sets to prevent gaming
Intelligence Index v4.2 changelog:
+ AA-Briefcase, our agentic knowledge work evaluation with a private test set
+ @HelloSurgeAI's GDP.pdf, long context document reasoning across 4,592 PDF pages
- GPQA Diamond, an exceptional scientific reasoning evaluation that has now been saturated
… plus greater weighting on held-out test sets to prevent gaming, and grading infrastructure upgrades to increase robustness
This update brings the Index closer to real-world use cases with more challenging, complex and realistic tasks and private test sets to prevent gaming. We have been planning and building elements of Index v5 for months - it’s been 8 months since we launched Index v4 in January.
We have deliberately held back updates to keep the Index stable through recent major model launches. However, with the frontier moving so quickly in the past weeks, we feel it is important to deliver an immediate interim update to ensure our Index remains as relevant and useful as ever to users.
Beyond this interim update, our team is hard at work on v5 of the Index. We are planning more incremental releases in the near future. Stay tuned!
Intelligence Index v4.2 changes in detail:
➤ Adding AA-Briefcase: Our in-house evaluation with a private held-out test set, AA-Briefcase tests models on realistic agentic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work.
➤ Adding GDP.pdf: Created by @HelloSurgeAI, GDP.pdf evaluates single-turn professional document reasoning across 100 PDFs and ten domains. Models must synthesize evidence distributed across 4,592 pages, including text, tables, charts, footnotes, and exclusions. Responses are graded against 1,275 expert-authored atomic criteria; the headline All-pass Rate credits a task only when every criterion is satisfied.
➤ Weighting to measure real-world use and prevent gaming: 40% of our Index weighting is now private, held-out test sets - double the figure from v4.1. Held-out data includes AA-Briefcase, AA-Omniscience, and solutions for CritPt. This reduces the ability for labs to game evaluations. The held-out percentage will increase further in Index v5.
➤ Improving our grading infrastructure: In AA-LCR v1.1, we have added a grading system prompt and corrected errors and ambiguities in answer keys, improving scoring accuracy. For GDPval-AA v2 and AA-Briefcase, we have improved our sampling and re-anchored the Elo scale, making ratings more stable as new models are added. For SciCode we have improved robustness of grading sandboxes to ensure slow but correct code does not count as a failure.
Key results:
➤ Anthropic and OpenAI lead the Index: Anthropic’s Claude Fable 5.1 leads the Index, followed by OpenAI’s GPT-6 Astra, which shows a 4pt gain over GPT-5.6 Sol. Meta is the third-ranked lab on the leaderboard, followed by SpaceXAI, Moonshot/Kimi, Z AI, and Google
➤ Cost per Task Pareto frontier shared by four labs: Anthropic, OpenAI, Meta and Z AI occupy the updated Cost per Task Pareto frontier
➤ GPT-6 Astra dominates the output token Pareto frontier: GPT-6 Astra is more token efficient than almost every other model near the intelligence frontier, with Claude Fable 5.1, Grok 4.5 and Gemini 3.5 Flash-Lite at either end of the curve (excludes models below 25 on the Index)
RT @echen: taste isn't a reasoning problem. you can't rubric your way from "correct" to "great."
great writing, design, even great science…
Surge AI retweeted
it's tuesday morning. you have a stack of work in front of you.
read your email. understand the chart from the DS team. tie it back to that Slack thread from last week. send your VP a rec by noon.
which model do you choose?
we built Tuesday: Frontier Work Index to answer that.
Surge AI retweeted
New transfer paper from our team:
We post-trained a model on Office Work (spreadsheets, docs, general tools).
Zero code, but the model also got better at Software Engineering: +5.8pp on SWE-Bench Pro, from just a couple hundred examples.
An analogy I like: imagine a junior engineer who takes a year off to plan weddings. When they get back, they're suddenly staff level 😎 Why?
— They learned how to turn "make it magical!" into a concrete plan with dates, vendors, and budget. (Transfer: your PM's vague spec gets turned into a concrete set of PRs.)
— They learned how a last-minute change in weather breaks the caterer's plans. (Transfer: a change to one module can break the consumers that depend on it.)
When we compared model trajectories before and after training, we saw the same 4 improvements across both office work and coding, in a capability we call Goal-Directed Execution:
1. Forming the right local goals
2. Building an accurate picture of the environment
3. Staying faithful to higher-level objectives
4. Verifying that goals were achieved
Surge AI retweeted
we trained a model on our coding dataset. +12.0pp on Terminal-Bench 2.0
we trained a model on our enterprise agents dataset. +9.6pp on Toolathlon
we trained a model on our STEM dataset. +9.3pp on FrontierScience
even better, some of our non-coding datasets improved coding anyway (+6pp on SWE-Bench Pro), without containing a single line of code
every one of these came from training on a fifth of the data. the full versions are 5x larger and now available off the shelf
