human-in-the-loop data infra | Prof, Ohio State CS | Co-founder, @thesteamfactory, @hackohio, @mobikitinc (acq.)
Columbus, OH
Joined March 2008
- Tweets300
- Following3.6K
- Followers3.1K
- Likes7.8K
Surprising results from @OpenAI 's new Terra release. It shouldn't be this good! Also separately, adding Clef into the benchmark reveals a new Pareto frontier: decisions are a very cost effective pathway for physical world understanding! (work led by @PerraultAndrew )
Thank you for confirming this! We are seeing the same thing with RealityBench. Still behind on our acceptability metrics, but very surprised to see Argon / Fable / Astra performance at such a strong price point. relantic.com/realitybench
@thepaulthomson
Terra = massive cost / quality unlock & Pareto inversion for visual perception, will change business models for many in physical AI. Compliments to the chef.
Arnab Nandi ⭐️ retweeted
Since the first release of RealityBench just 2.5 weeks ago, we've seen a substantial upheaval among the top models. First Gemini 4 Argon came out and set or matched top performance, but for 50%-6x cheaper. Then we saw new strong releases from OpenAI in 6.1 Sol and now 6.1 Terra.
Surprising results from @OpenAI 's new Terra release. It shouldn't be this good! Also separately, adding Clef into the benchmark reveals a new Pareto frontier: decisions are a very cost effective pathway for physical world understanding! (work led by @PerraultAndrew )
Arnab Nandi ⭐️ retweeted
We ran @Elorian's new Glimpse model on RealityBench's physical work tasks: it reads labels as well as Argon, but has gaps in fine-grained visual reasoning: tape measures, lag bolts. Also, reasoning adds up quickly: our run cost $18 for a score that Gemma 4 31B matches for $0.70.
Arnab Nandi ⭐️ retweeted
Why build worlds? A work photo is the first mile of AI: if the capture is wrong, no model can reason its way back.
But real field data is locked away and barely labeled: thousands of photos per job, one label for the whole packet. Nobody knows what a model should see in each.
Sharing something we've been working on! @relanticai's RealityBench is a portfolio of 3D worlds which we used to evaluate 30+ frontier models on physical work tasks, incl. Argon, Astra, Fable, and Muse Wonder. Because these are 3D worlds, we can "bend" reality for new GT data.
The worlds are simulators, not image sets: built to extend to new tasks. Early access is on Hugging Face and relantic.com/realitybench Check out our video below.
Arnab Nandi ⭐️ retweeted
Task-specific routing can pay off hugely. For example, GPT-6 Astra, a top model, read only 46% of VINs correctly, whereas even Sonnet 5 gets 86%.
Arnab Nandi ⭐️ retweeted
The pace of improvement has been shocking. I half-expected that the focus on coding and RSI would mean that progress on vision had slowed. Instead, it seems likely that VLMs will be clearly superhuman in performance and cost in the next year or so.
Arnab Nandi ⭐️ retweeted
The models have a key structural advantage: superhuman attention. When you are looking at photos for details for a long time, it is easy to miss some.
Arnab Nandi ⭐️ retweeted
What did we learn about current models? The models that performed best are essentially price and performance competitive with a single human rater on average.
Arnab Nandi ⭐️ retweeted
We split the tasks into two categories: acceptability, where a VLM must judge if an image meets certain properties (e.g., car is in frame, license plate text is legible) and documentation, where the VLM must extract info from the scene (e.g., read a measuring tape).
Arnab Nandi ⭐️ retweeted
We build programmatic scorers for each. Humans disagree substantially on these tasks. We only release environments and scorers where the scorer closely matches the human majority in our testing.
Arnab Nandi ⭐️ retweeted
In some cases, we can compare VLM performance in these environments to real-world data from the same tasks. Model rankings transfer (Spearman ρ ≥ 0.63), suggesting the simulated environments capture what makes these tasks easy or hard for models.
Arnab Nandi ⭐️ retweeted
We created simulated environments in Blender for benchmarking VLMs on real world physical work perception and reasoning tasks. Think construction and maintenance of houses and cars.
Excited to share what we've been working on! @relanticai #RealityBench uses 3D worlds to evaluate 30+ frontier models on physical work understanding.
Arnab Nandi ⭐️ retweeted
Request early access to environments and images here: relantic.com/realitybench. Paper: arxiv.org/abs/2610.14397. Joint work with @arnabdotorg, @anmoltakiar and Dominik Winecki at @relanticai.
.@cbinnig should have titled this paper "welcome to the jungle"
"real-world DBs = highly normalized, even simple NL questions to require 4 joins/query, vs Spider (0.5J/query) ...state-of-the-art models experience up to a 70% drop in execution accuracy" openreview.net/pdf?id=NZLm4v…
Couldn't agree more. Recent "AI interaction" paradigms unlocked a huge wave of productivity. However, it's time to reclaim what we left behind.
One big regret about the LLM and agents wave is that subsecond interaction has taken a backseat. After decades of building towards “speed of thought” flow states, we’re now back to “brb, Claude’s compiling”.
Nice "Databases Wrapped" by @andy_pavlo. MCP is awesome, but MCP is also the fastest accelerating tech debt that databases have had to endure since MongoDB. Looking forward to the new wave of security, data loss prevention, access control, and perf recovery offerings for MCP.
Here is my latest article on the world of databases: cs.cmu.edu/~pavlo/blog/2026/…
All the hot topics from the last year:
• More Postgres action!
• MCP for everyone!
• MongoDB gets litigious with FerretDB!
• File formats!
• Market movements!
• The richest person in the world!
