As models improve, how should the data brief evolve?
At OrphLux, we work with frontier researchers and data experts to keep up with our clients’ changing needs.
Four priorities keep coming up — expert judgment, realistic tasks, real-world coverage, and clear evaluation criteria.
🤖 Made with AI
4
41
Higher scores at less than a third of the cost per task!
That makes testing on real workflows a lot more practical.
Next up is seeing how much expert review and rework the outputs still need.
Replying to @OpenAI
GPT-6.2 Astra—a frontier model built for your hardest projects. It sets new highs on evals across professional work and scientific research at less than a third of the cost per task of GPT‑6 Astra.
You’ll see the difference in things like slideshows, websites, and 3D designs.
27
🍖A wonderful BBQ night in Los Altos Hills with researchers from @OpenAI , @GoogleDeepMind , @xai , @MicrosoftAI, and @AMD.
Compute efficiency. Infrastructure for agents. AI helping advance AI research.
Happy to be a sponsor. Thanks to @BFC_Global for bringing us together!
Independent review is more than useful when experts can inspect the cases behind a score! For data providers like us, keeping test data linked to grading decisions and model versions gives reviewers a clearer basis for finding gaps.
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards.
These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems.
openai.com/index/our-first-p…
47
A real factory task is a useful starting point. Shift the part or test recovery from a missed grasp to see how well the approach holds up.
We gave Claude Fable 5.2 a real task from a precision-machining factory and a robot arm.
It got the job done, writing its own code. No teleop, no demos, no task-specific training.
Agentic robots are where physical AI starts doing real work !
37
Excited to see GR00T 2 learn from human demonstrations. Capturing how objects change after each action gives teams a clearer way to evaluate what the robot learned.
Meet GR00T 2, our most advanced, open System One model for robotics research. It learns skills from human demonstrations and predicts how its actions will change the physical world.
Those skills can carry across different robot bodies, reducing the need to teach each robot every task from scratch.
Watch GR00T 2 put that research into motion. 🧵
1
33
Excited to see Atlas open to everyone. We’d love to explore how well a scene holds together as the camera moves. Reference data showing the same space from multiple viewpoints could help evaluate that consistency.
Atlas is now available to everyone!! I’m very excited to see what users can build with our newest world model! 🤩🌎
38
The interesting question is what robotics research becomes. As models improve, understanding where they still fail and which new demonstrations actually help could become an even bigger part of the work.
3
161
Building the engine also opens up a data opportunity. Varying lighting, motion and camera paths could create controlled test scenes. For model evaluation, the useful part is knowing exactly what changed between samples.
Googlers have been testing and exploring Gemini 4 Carbon for the past few weeks. Here are a few of our favorites 🧵
To capture the light and motion of this 3D underwater swarm, @fofrAI used Carbon to build a custom CUDA and C++ visual effects engine from scratch.
51
As robots learn new skills faster, choosing the next demonstrations matters more. The same task with a different tool, object or workspace can reveal gaps that more repetitions of a familiar setup might miss. That’s a useful starting point for the next data collection.
Introducing Gemini Robotics 2.5, our most capable vision-language-action (VLA) model to date. 🤖
It increases both the breadth of skills a robot can learn and the speed at which it acquires them — helping robots step out of the lab and into real-world environments.
Learn more about Gemini Robotics 2.5, now rolling out to our robotics partners 🧵
1
103
OpenAI’s finding that CoT controllability can pass from teacher to student adds a question for synthetic data evaluation: alongside answer quality, what behaviors are we passing on? Teacher selection deserves its own checks.
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle?
We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
47

















