Vik Paruchuri (@VikParuchuri) profile picture

Open source AI. Founder of @datalabto Past: founded @dataquestio

Brooklyn,NY
Joined June 2012
Benchmarking is hard, even for experts. And most benchmarks are made by vendors who want to rank #1, not help you judge things fairly. I gave a talk on how you can stress-test benchmarks yourself.
3
1
2
40
3,073
The Datalab team and I have done a lot of work to audit benchmarks and make the field better. For example, here is some detailed analysis on other benchmarks - datalab.to/blog/why-we-built… . If we can help you with evals, feel free to reach out.
1
1
248
We shipped Chandra 2.7! Nothing massive, just steady OCR improvement: - Better tables: balanced 91.6 → 94.2 on OmniParseBench - Better on non-English and rotated pages
9
17
132
4,818
Other gains: - Rotated pages: 81.4% → 87.3% of tests pass (balanced) - Forms: 92.3% → 93.7% (accurate) - Non-English pages: 86.7% → 87.9% (accurate)
1
8
544
There are so many flawed OCR benchmarks, and more come out every week. I gave a talk on how to evaluate them in @HamelHusain's course. Here are common issues, how to spot them, and a guide on evaluation 🧵
12
13
5
113
15,713
Many benchmarks ship worse settings/defaults for other providers or have major bugs. On ExtractBench (LlamaIndex), our score went from 64.5% to 93.6% when we fixed the adapter. It had counted our confidence and citation fields as wrong values.
1
3
740
Here's the full guide: how to check any benchmark in 5 minutes, and how to run an eval on your own documents - datalab.to/blog/how-to-evalu… .
2
1
1
18
2,551
Datalab fast mode OCR is now more accurate than most frontier models and APIs, including Opus 5.5, GPT-6.1 Sol, and Mistral OCR. Cost: $1 per 1,000 pages, which is 1/4 to 1/68 their prices.
15
10
114
4,465
We measured accuracy with OmniParseBench, our fair parsing benchmark - github.com/datalab-to/omni_p… . The only models that score higher are Gemini 3.8 Flash and Gemini 4 Argon/Carbon, at 37x the cost or more, and 6x or more slower.
1
1
1
320
We're continuing to improve accuracy, speed and cost across every mode. More soon.
251