shaped retweeted
An independent audit found that the OSWorld 2.0 desktop-agent benchmark gives full credit to runs that clearly failed. Reviewers hand-checked all 108 tasks and found 43 issues (18 major, 25 minor) out of 65 clean tasks, meaning headline leaderboard scores like 72.6 may not reflect real performance. The audit, OSWorld 2.0 Verified, publishes task-level evidence for each finding. parsewave.ai/evaluation-sign…
shaped retweeted
Happy to see that our RL work is helping to the move the frontier 🫡
shaped retweeted
hating on leopold is so weird. he predicted so many things right, put out a timeline thesis, bet all in on it, and priced in things no one else did. now there is always a chance something doesn't work out
but it literally did work out he is up 80% this year
shaped retweeted
For my eval-maxxing nerds out there, good friends of mine are running a series called "strange evals", you can benchmaxx now on anything. If in SF swing by! luma.com/lvqbs1mo
shaped retweeted
No major benchmark is designed for COBOL, Fortran, or Assembly - the languages powering trillions in transactions and infrastructure that must be modernized or risk catastrophic failure.
We built Legacy-Bench to measure frontier agents on the code the world actually runs on.
shaped retweeted
My AI investment thesis is that every AI application startup is likely to be crushed by rapid expansion of the foundational model providers.
App functionality will be added to the foundational models' offerings, because the big players aren't slow incumbents (it is wrong to apply the analogy of "fast startup, slow incumbent" here), they are just big. Far more so than with any other prior new technology, there is a massive and fast-moving wave that obsoletes every new app almost as fast as it can be invented. There is almost no time to build a company and scale it.
There are two ways AI application startup founders can make money:
- Make a flash-in-the-pan app that generates a ton of cash and bank the cash (my estimate is that you have about 12-18 months cashflow generation)
- Make a good enough app that you get acquired by one of the big players for sufficient equity
The situation is highly unstable - we don't know if it's going to crash or go to the moon but both scenarios make it very unlikely that any AI application startup will independently become a generational supercompany (baseline odds are low to begin with).
The best odds are finding an application niche in a highly specialized field with extremely unique and specific data barriers, ideally ones relating to real atoms (hardware or world-related) data and not software/finance.
shaped retweeted
Self-Improving Embodied Foundation Models
Project: sites.google.com/view/mfa-se…
Paper: arxiv.org/abs/2509.15155
New paper from Google Deepmind on a two-stage post-training approach for robotics, applying Supervised Fine-Tuning (SFT) and Self-Improvement, inspired by the success of RL stage in finetuning large language models.
- Supervised Fine-Tuning (SFT): finetune multimodel foundation model with (1) behavior cloning (2) steps-to-go prediction objectives.
- Self-Improvement: use steps-to-go prediction to extract reward function and success detector, which enable a fleet of robots to do online RL to practice downstream tasks with minimal human supervision.
- Authors demonstrate through experiments that SFT+Self-Improvement is more sample-efficient and leads to better performance than purely scaling imitation data collection for supervised learning.
- Authors also demonstrate this combination can enable autonomously acquiring new skills that generalize beyond the tasks covered in the imitation learning datasets.
shaped retweeted
Cool research paper from Google.
This is what clever context engineering looks like.
It proposes Tool-Use-Mixture (TUMIX), leveraging diverse tool-use strategies to improve reasoning.
This work shows how to get better reasoning from LLMs by running a bunch of diverse agents (text-only, code, search, etc.) in parallel and letting them share notes across a few rounds. Instead of brute-forcing more samples, it mixes strategies, stops when confident, and ends up both more accurate and cheaper.
Mix different agents, not just more of one: They ran 15 different agent styles (CoT, code execution, web search, guided variants). Each agent sees both the question and other agents’ past answers, then tries again. This back-and-forth makes the group smarter than any single agent.
Stop early, save cost: More rounds don’t always help. Too much refinement can kill diversity. They use an LLM-judge to decide when to stop. That keeps accuracy high while cutting costs almost in half.
Better than existing methods: Compared with other tool-augmented scaling tricks, TUMIX consistently scores higher on tough reasoning benchmarks (HLE, GPQA-Diamond, AIME). For Gemini-2.5 Pro, it pushed HLE to 34.1%, which is a notable gain.
Diversity is the secret sauce: Combining text, code, and search agents beats repeatedly sampling the best single agent. More diverse tool use = more chances to land on the right reasoning path.
Auto-agent design: They even had the LLM generate new agent types and mixed those in, which boosted results further. The sweet spot was around 12–15 different agent styles in the mix.
arxiv.org/abs/2510.01279
