Ramp Labs (@RampLabs) header
Ramp Labs (@RampLabs) profile picture
Joined August 2025
Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)Photo by Ramp Labs (@RampLabs)
We’re publishing Ramp SWE-Bench v2, featuring 500 tasks built from work shipped to production by over 200 of our engineers, more rigorous verification, and frontier models running in their native harnesses. labs.ramp.com/swebench
36
21
34
457
149,382
Our goal is to measure how models perform on the tasks Ramp engineers ask AI to do today. As models become more capable and engineers delegate increasingly challenging work, this benchmark will evolve alongside them. Getting this measurement right means rethinking how to capture engineering judgement beyond deterministic tests. Here’s how we did it → builders.ramp.com/post/build…
2
20
4,448
Explore the full leaderboard, read our methodology, and compare models and harnesses head to head → labs.ramp.com/swebench
1
16
3,365
Claude Fable 5.2 is the top Anthropic model on Ramp Accounting Bench. It scores 52.2%, ahead of Opus 5.5 (50.5%). Only Gemini 4 Argon scores higher (61.1%). It solves more tasks on the first try than Opus 5.5 (13.9% vs 10.9%) and runs 2.8x faster than Argon (6 vs 16.7 min).
18
8
3
313
28,438
Reframing cartridge training as a self-play game cuts the training steps needed by more than half, just by asking better questions. This post continues our series on cartridges, compressed KV caches of long documents. Here we focus on self-study, the method used to train them. Self-study teaches a model reading the cartridge to answer the way it would with the full document in context. Currently, its questions come from fixed, generic prompts, so many of them ask about things the cartridge already knows.
8
9
1
130
7,260
On a long patient record from LongHealth, adaptive self-study matches baseline self-study's final accuracy by 5,120 steps, under half. It then keeps improving, finishing at 80% vs 76.6%. But optimizing for difficulty alone collapses onto one part of the document. Read about how we solve this with a bandit and more below.
1
3
302
It checks its work much earlier. Argon makes its first check 44% of the way through an attempt. For Opus 5.5, it's 77%.
1
1
11
2,678
Gemini 4 Argon is the new top model on Ramp Accounting Bench. It scores 61.1%, 10.6 points ahead of Opus 5.5. Its lead is largest on the hardest tasks.
26
33
12
665
75,796
We had early access to @NVIDIAAI Nemotron 3.5 Super VL and post-trained it for one of our highest-volume workflows: receipt extraction. One epoch of supervised fine-tuning on 10,000 examples made it competitive with leading open and closed-source models on our receipt benchmark.
5
5
8
151
17,003
At 750 requests/min, the post-trained model delivered ~68% lower p50 and p95 latency than DeepSeek V4.1 Flash. At 1,500 requests/min, receipt throughput was ~31% higher.
1
5
548
Model selection is as much a systems problem as a quality one: latency, throughput, and cost matter. See more of our work at labs.ramp.com/research
5
439