Our goal is to measure how models perform on the tasks Ramp engineers ask AI to do today. As models become more capable and engineers delegate increasingly challenging work, this benchmark will evolve alongside them.
Getting this measurement right means rethinking how to capture engineering judgement beyond deterministic tests. Here’s how we did it → builders.ramp.com/post/build…
2
20
4,448
Explore the full leaderboard, read our methodology, and compare models and harnesses head to head → labs.ramp.com/swebench
1
16
3,365
Reframing cartridge training as a self-play game cuts the training steps needed by more than half, just by asking better questions.
This post continues our series on cartridges, compressed KV caches of long documents. Here we focus on self-study, the method used to train them. Self-study teaches a model reading the cartridge to answer the way it would with the full document in context. Currently, its questions come from fixed, generic prompts, so many of them ask about things the cartridge already knows.
On a long patient record from LongHealth, adaptive self-study matches baseline self-study's final accuracy by 5,120 steps, under half. It then keeps improving, finishing at 80% vs 76.6%. But optimizing for difficulty alone collapses onto one part of the document. Read about how we solve this with a bandit and more below.
1
3
302
We had early access to @NVIDIAAI Nemotron 3.5 Super VL and post-trained it for one of our highest-volume workflows: receipt extraction.
One epoch of supervised fine-tuning on 10,000 examples made it competitive with leading open and closed-source models on our receipt benchmark.
Model selection is as much a systems problem as a quality one: latency, throughput, and cost matter. See more of our work at labs.ramp.com/research
5
439








