Public LLM Evaluation // https://xitter.cf/t.co/rNS39rhVQ9 @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder
San Francisco, CA
Joined March 2024
- Tweets2.3K
- Following280
- Followers28.6K
- Likes826
Pinned Tweet
For one weekend, we are benchmaxxing on purpose!
Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.
Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.
Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.
Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
It costs $0.21 per 1,000 claims and answers in under a second. Haiku 5.5, the cheapest of those language models, costs 12x more and takes about 5 seconds.
Liquid's earlier d1 missed 26 claims, and most of those were subtle contradictions, where one small wording change makes the claim wrong. d1-ultra got every one of them. It costs about twice what d1 does.
Vals AI retweeted
if anyone's curious- github.com/vals-ai/Valkyrie is the framework we use to create and run evals!
Claude Opus 5.5 leads the index at 65.6%, with Gemini 4 Argon (64.9%) within the margin of error.
However, no model leads every component. Opus 5.5 tops Finance Agent and Tax Agent Bench, and Claude Sonnet 5.5 tops Excel Modeling. Even the leaders fail more than half of the tax questions.
Finance reshuffles the leaderboard: Gemini 3.8 Flash and Kimi K3 place far higher here than on the Vals Index, while GPT-6 Astra and GPT-6.1 Sol place far lower.
Opus 5.5 costs $11.67 per test run, while Wonder Full is 1.0 points behind at $3.14, 27% of the cost. DeepSeek V4.1 Flash 1017 is 4.7 points behind at $0.42, about 4% of the cost.
Full results and methodology: vals.ai/benchmarks/finance_i…
It is also quick: 0.55 seconds per search on average. Search time is about a tenth of task wall time; the rest is model inference.
Congratulations to the @KeenableAI team. For full results, visit: vals.ai/benchmarks/web_searc…
