Aman Khan (@amankhan) header
Aman Khan (@amankhan) profile picture

AI PM @google on evals, benchmarks | ex @arizeai views my own

Joined December 2011
Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)Photo by Aman Khan (@amankhan)
I promise I’m not just saying this because I work at @Google But Gemini generated analysis and tokens are just more pleasant to read than other LLM-speak (I.e “claudese”)
48
7
10
473
43,994
Aman Khan retweeted
Build an open-source autonomous agent harness, pair it with one of 8 Google-sponsored foundation models, and tackle Physical-Engineering's Last Exam (PLE), a benchmark of 85 expert-reviewed, long-horizon tasks across structural, civil, electrical and mechanical engineering. Join the Agent Harness on Physical-Engineering's Last Exam competition, hosted by @Google! 💰 Prize Pool: $400,000 ⏰ Entry Deadline: January 5th, 2027
29
108
3
1,448
64,698
Aman Khan retweeted
Benchmarks tell you how models score. Arenas let you judge and see how they stack up. Introducing Kaggle Arenas: head-to-head matchups where AI models take on the same challenge, and you decide which one comes out on top. From 3D modeling to creative design to data visualization, see what today's models can really do. Every vote helps shape the leaderboard. Start voting: kaggle.com/arenas
26
16
6
137
15,112
as models get better, writing good evals gets harder and more important.
2
1
25
1,327
Aman Khan retweeted
Carbon solves more of the task frontier: it passed at least once on 14 of 20 tasks, vs 11 for Claude Opus 5.5 and 12 for GPT-6 Astra. It’s the only one of the three to solve the Stripe and Mastodon clones.
2
17
316
ground truth. reskilling/upskilling/learning how to use AI to your advantage remains an incredible opportunity
STATE OF THE TECH JOB MARKET IN 2026 Twice a year, I dig into trends in hiring and job openings. This is the fifth report I've done, and the most surprising yet: 1. PMs are winning in the AI era 2. Engineering demand has stalled 3. Design openings are at a record low 4. Every role now requires AI skills 5. Tech jobs are concentrating in the Bay Area 6. Remote roles are at a record low 7. The tech job market is growing fast, but mostly outside of software (gasp!) Full report here: lennysnewsletter.com/p/state…
1
12
576
Aman Khan retweeted
“Both of my parents got diagnosed with two cancers each within a week of each other.” From @AshleyYuki, VP of Product at @OpenAI: “I just decided to take a month [off work]. It was one of the best things I could have done in that moment to care for my parents. You also have to care for yourself as a caregiver. We don’t talk about that enough. It’s still a really big cognitive load, emotional load, just pure time load to be a caregiver. You have to make sure you’re budgeting time for your own mental health.” 📌 Watch our full episode here: youtu.be/pR7NCYKsA3o
“Both of my parents each got diagnosed with cancer within a week of each other.” Ashley is the VP of Product at @OpenAI for Health and this is a very special episode for me. We talked about how we both use AI to help our parents fight cancer — including preparing for doctor visits, researching treatments, and coping with our emotions as caregivers. Topics we discussed: → Do doctors mind when patients bring AI research? → How ChatGPT caught a risky treatment before it was given → How to make time for your parents with a demanding job Some quotes from Ashley: “I always felt like there was no more time in the day. Then someone gets sick and you realize what you can make time for and what can drop.” “You also have to care for yourself as a caregiver. You have to make sure you’re budgeting time for your own mental health.” “The treatment she was about to be given would have had a really bad contraindication. She narrowly avoided it because she ran it through Chat.” 📌 Watch now: youtu.be/pR7NCYKsA3o Thanks to our sponsors: @meetgranola: AI meeting notes that don’t suck granola.ai/peter @RiversidedotFM: All-in-one AI studio for podcasts and video creators.riverside.com/Peter…
5
8
74
16,019
This is indeed a problem and something we are thinking more about
jammed with @HamelHusain a bit and since then, i cannot unsee of the state of evals as being similar to the military-industrial complex
3
19
1,523
This one hit close to home - AI is incredible at leveling the playing field (being a thinking partner) in discussions + decisions where information is asymmetric and imperfect. Proud of @petergyang for shining a light here
“Both of my parents each got diagnosed with cancer within a week of each other.” Ashley is the VP of Product at @OpenAI for Health and this is a very special episode for me. We talked about how we both use AI to help our parents fight cancer — including preparing for doctor visits, researching treatments, and coping with our emotions as caregivers. Topics we discussed: → Do doctors mind when patients bring AI research? → How ChatGPT caught a risky treatment before it was given → How to make time for your parents with a demanding job Some quotes from Ashley: “I always felt like there was no more time in the day. Then someone gets sick and you realize what you can make time for and what can drop.” “You also have to care for yourself as a caregiver. You have to make sure you’re budgeting time for your own mental health.” “The treatment she was about to be given would have had a really bad contraindication. She narrowly avoided it because she ran it through Chat.” 📌 Watch now: youtu.be/pR7NCYKsA3o Thanks to our sponsors: @meetgranola: AI meeting notes that don’t suck granola.ai/peter @RiversidedotFM: All-in-one AI studio for podcasts and video creators.riverside.com/Peter…
1
11
1,018
find your lane lean into it
5
500
BREAKING: SPY drops to lowest level since Tuesday
2
493
nice
Models by @GoogleDeepMind scale better on cost than their rivals. Gemini 4 Carbon is the prime example. As tasks get more demanding, every rival gets more expensive relative to Carbon. For a typical task, GPT-6 Astra by @OpenAI costs 3.4× as much, Claude Opus 5.5 by @AnthropicAI costs 3.9× as much, and GPT-6.1 Sol is the cheaper option at about half of Carbon’s cost (0.54×). Across the 5% most expensive sessions, Carbon’s position improves against all three. Opus edges up to 4.2×, Astra’s cost roughly doubles relative to Carbon, and Sol’s discount shrinks. This in turn can translate into meaningful savings across real-world tasks such as building an Android security app, one of many long-horizon tasks that our Agent Arena users delegate to their agents.
3
467
(Hill) climb some benchmarks (walls) with this crew
2
241
Apparently this is less of a “hot take” than I realized - a lot of people agree! If you haven’t tried Gemini for writing, give it a shot and let me know what you think - would love to pass feedback on to the team
I promise I’m not just saying this because I work at @Google But Gemini generated analysis and tokens are just more pleasant to read than other LLM-speak (I.e “claudese”)
1
25
2,526
Such a high signal event. Many of the top people in the AI benchmark/eval field, both on stage and off stage. @kaggle Bench Summit
1
1
1
31
1,655
This is worth following!
Bench Summit by Kaggle has officially kicked-off! Trust in benchmarks is eroding. The path forward starts here. We started strong with @janiejgu sharing key insights on the problems with benchmarks today and Kaggle’s mission to build trust in evals and help the world discover what works and doesn’t work in AI. During the session, she shared four ways benchmarks fail today: 1. Relevance: Many public benchmarks don’t resemble what your agents do in production. 2. Saturation: Evals go stale within a release cycle. 3. Verification: Everyone has their own implementation of the same benchmarks, meaning scores vary wildly due to infrastructure, not model quality. 4. Infrastructure: Evals infra is hard to build correctly, and everyone rolls their own. To get numbers you can actually trust, Janie highlighted four essentials: 1. Real workflows as the foundation for benchmarks 2. Continuous benchmarks from crowdsourced tasks, arenas, and production data 3. Rigorous and transparent methodology audited by the community experts 4. Eval infrastructure that everyone can use out of the box Here’s a sneak peak of the early moments:
11
547
Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago. Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
1,237
1,155
862
15,862
1,310,137
superhuman intelligence is literally VC subsidized what an insane time to be alive
1
3
434
Aman Khan retweeted
Abundant is partnering with @baseten to empower customers to own their capabilities. Excited to leverage our experience working with every frontier lab to build environments tailored to the customers' unique needs.
3
13
5
64
7,447