Public LLM Evaluation // https://xitter.cf/t.co/rNS39rhVQ9 @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder
San Francisco, CA
Joined March 2024
- Tweets2.3K
- Following280
- Followers28.6K
- Likes826
Pinned Tweet
For one weekend, we are benchmaxxing on purpose!
Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.
Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.
Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.
Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
Vals AI retweeted
if anyone's curious- github.com/vals-ai/Valkyrie is the framework we use to create and run evals!
Claude Opus 5.5 leads the index at 65.6%, with Gemini 4 Argon (64.9%) within the margin of error.
However, no model leads every component. Opus 5.5 tops Finance Agent and Tax Agent Bench, and Claude Sonnet 5.5 tops Excel Modeling. Even the leaders fail more than half of the tax questions.
Finance reshuffles the leaderboard: Gemini 3.8 Flash and Kimi K3 place far higher here than on the Vals Index, while GPT-6 Astra and GPT-6.1 Sol place far lower.
Opus 5.5 costs $11.67 per test run, while Wonder Full is 1.0 points behind at $3.14, 27% of the cost. DeepSeek V4.1 Flash 1017 is 4.7 points behind at $0.42, about 4% of the cost.
Full results and methodology: vals.ai/benchmarks/finance_i…
It is also quick: 0.55 seconds per search on average. Search time is about a tenth of task wall time; the rest is model inference.
Congratulations to the @KeenableAI team. For full results, visit: vals.ai/benchmarks/web_searc…
We tested 18 AI chatbots on teen safety with SAFE-Teen, the first teen safety benchmark built with clinicians that follows whole conversations instead of grading the first answer.
Every chatbot failed. 54.0% of conversations had at least one safety failure. The range was wide: 14.0% for GPT-6.1 Sol, 62.4% for Gemini 3.8 Flash.
Pushback is where boundaries break. When a chatbot set a limit and the teen pushed back, failures rose from 22.0% to 270%.
Mental health diagnosis went from 43.0% to 54.7%. Help with substance misuse went from 27.6% to 39.5%.
A correct first answer doesn’t count for much if the model gives in a few turns later.
5/ One caveat: we tested through developer APIs, not the consumer apps, which can add their own filters, age checks and parental controls. Our results show how the underlying models behave.
Read our full blog post: vals.ai/blogs/teen-safety
We ran Beam through Reflection's own API with temperature 0.7 and top_p 0.99, It has a 1M token context window, 128K max output, and is free to the public!
Congrats @reflection_ai. Full results at vals.ai/models/reflection_Be…
