
Open source AI. Founder of @datalabto Past: founded @dataquestio
Brooklyn,NY
Joined June 2012
- Tweets2.3K
- Following191
- Followers18.6K
- Likes2K
Benchmarking is hard, even for experts. And most benchmarks are made by vendors who want to rank #1, not help you judge things fairly.
I gave a talk on how you can stress-test benchmarks yourself.
The Datalab team and I have done a lot of work to audit benchmarks and make the field better. For example, here is some detailed analysis on other benchmarks - datalab.to/blog/why-we-built… .
If we can help you with evals, feel free to reach out.
1
1
248
The written version of the talk is here - datalab.to/blog/how-to-evalu… .
216
We shipped Chandra 2.7!
Nothing massive, just steady OCR improvement:
- Better tables: balanced 91.6 → 94.2 on OmniParseBench
- Better on non-English and rotated pages
Other gains:
- Rotated pages: 81.4% → 87.3% of tests pass (balanced)
- Forms: 92.3% → 93.7% (accurate)
- Non-English pages: 86.7% → 87.9% (accurate)
1
8
544
Read more at datalab.to/blog/chandra-2-7 , and kudos to @hheiden12 .
You can try it on the playground at datalab.to/playground/docume… .
We will keep delivering the highest quality at the lowest latency/price. A lot more to come on this front soon.
7
462
There are so many flawed OCR benchmarks, and more come out every week.
I gave a talk on how to evaluate them in @HamelHusain's course. Here are common issues, how to spot them, and a guide on evaluation 🧵
Many benchmarks ship worse settings/defaults for other providers or have major bugs. On ExtractBench (LlamaIndex), our score went from 64.5% to 93.6% when we fixed the adapter. It had counted our confidence and citation fields as wrong values.
1
3
740
Here's the full guide: how to check any benchmark in 5 minutes, and how to run an eval on your own documents - datalab.to/blog/how-to-evalu… .
Datalab fast mode OCR is now more accurate than most frontier models and APIs, including Opus 5.5, GPT-6.1 Sol, and Mistral OCR.
Cost: $1 per 1,000 pages, which is 1/4 to 1/68 their prices.
We measured accuracy with OmniParseBench, our fair parsing benchmark - github.com/datalab-to/omni_p… .
The only models that score higher are Gemini 3.8 Flash and Gemini 4 Argon/Carbon, at 37x the cost or more, and 6x or more slower.



