Akshit (@akshitkr) header
Akshit (@akshitkr) profile picture

Research Engineer looking at data™️, evals @ProximalHQ prev: CS, Linguistics, MechInterp research @iiit_hyderabad

Bengaluru, IN
Joined February 2018
Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)Photo by Akshit (@akshitkr)
Looking at the data™️ was/is the most interesting part about FrontierSWE v2. We're still looking at them and keep finding exciting bits to talk about. Have talked about a few in the blog, and have released all traces, hope this leads to more ppl looking at data™️
We are releasing FrontierSWE v2, our updated ultra-long horizon coding benchmark V2 features an expanded task suite and improved methodology. We see large performance gaps between frontier models, with Claude Fable 5.1 leading by a wide margin
1
1
25
1,647
CHOOSE YOUR CHARACTER
297
13,309
304
92,267
971,785
One interesting bit of AI discourse: it's pretty broadly agreed that agents aren't great at software design. But...they're still good at getting tasks done. So, maybe software design doesn't matter? I think there's a lot of complexity baked into this question!
69
19
6
503
27,753
FrontierSWE mentioned! (biased sample)
I asked 56 people in AI which benchmarks they'd trust most to indicate if an LLM is frontier: 25 votes: Terminal-Bench 4 (@alexgshaw, @andykonwinski, @Mike_A_Merrill) 20 votes: Terminal-Bench Science (@StevenDillmann, @lschmidt3, @sanmikoyejo) 13 votes: FrontierCode (@penlume, @ybenpan, @denizbirlikci, @silasalberti) 10 votes: FrontierSWE (@evan_j_chu, @_rajanagarwal, @MatternJustus, @ardnoKhtnayhsiR, @sanket__mhatre, @akshitkr) 8 votes: Humanity's Last Exam (@longphann_, @hendrycks, Alice Gatti, @ziwen_h, @natliml) 7 votes: Vending-Bench (@axelbacklund, @lukaspet) 5 votes each: - AutomationBench (@danielwshepard, @SalimansRobin, @TweetAnnaMarie) - DeepSWE (@winkey_h, @charleyslee, @LeonardMainnet, @serenaa_ge) - PostTrainBench (@full__rank, @hrdkbhatnagar, @AmyPrb, @shiraeis, @karinanguyen, @MatthiasBethge, @maksym_andr) The most common comment was that people don't really trust benchmarks at all. 56 people voted. People couldn't pick a bench their company worked on, an index, or a human-voted board. Disclosure: on this list, I'm a small investor in Abundant, Cognition, Emulated, Fleet, Hillclimb, Latch, and Snorkel. I didn't vote. Counts are net: a few benchmarks received negative votes. Continued:
10
357
Akshit retweeted
Great paper on why SFT can generalize just as well as RL if you give it enough diverse training data. Like everyone else since DeepSeek-R1 dropped, I’ve been treating "RL generalizes, SFT memorizes" as gospel. Whenever SFT runs plateaued on out-of-domain reasoning, everyone just blame the algorithm. Turns out, we might have been running an unfair comparison the entire time. In standard setups, RL gets to explore a huge rollout budget per prompt while SFT gets trained on a tiny slice of data. So the authors decided to actually level the playing field on Qwen2.5 and Qwen3 models by testing two things: mixing data sources instead of just distilling from one giant teacher, and scaling up the SFT data to match RL's rollout budget. The results completely flip the usual narrative. A carefully mixed dataset easily beats single-teacher distillation, and matching RL's rollout budget lets SFT generalize just as well as RL. And when you combine both, SFT actually outperforms it. To figure out why, they used LLM annotations to inspect the rollouts and proved (both empirically and theoretically on shallow transformers) that larger data budgets expose SFT to rare, "tail" solution methods. Those uncommon reasoning paths are where the actual generalization signal lives. Honestly, for everyone tired of debugging finicky online RL pipelines, this is a huge relief. SFT wasn't broken, we were just starving it of diverse rollouts. Full paper PDF: arxiv.org/pdf/2611.14912
27
90
9
692
38,727
Akshit retweeted
bangers from the private notebooks of Ludwig Wittgenstein
34
219
42
2,426
122,090
Meet Gemini 4 Hydrogen: built for rapid coding, multimodal synthesis, and everyday workflows. It finishes tasks 2.5x faster than 3.8 Flash and surpasses Argon on the @ArtificialAnlys Intelligence Index.
472
629
518
7,527
1,578,051
Akshit retweeted
In tiny Rajpur Village, Dehradun, 78 men were marched off to join WWI on behalf of the British Empire No one seems to know how many men returned home safe
20
130
3
857
15,947
Akshit retweeted
Claude Fable 5.2 ranks #1 on FrontierSWE. Fable 5.2 achieves a score of 69.9%, outperforming GPT-6 Astra (65.5%), Opus 5.5 (62.3%) and Gemini 4 Carbon (61.2%)
13
15
6
137
12,601
This is a fantastic guide to Kant’s Critique of Pure Reason. It also teaches you some of the basic philosophical concepts Kant relied on when writing the Critique, which new readers may not be familiar with. It has diagrams that make the more abstract ideas easier to follow.
8
64
8
705
18,565
We asked our AI one question about Liam Livingstone: “How do we bowl to him?” → The ball that shuts him down is the left-arm spinner’s stock ball: −44.5% efficiency against his own normal game, over 333 balls. Bowl him a slower ball and he feasts: +44.2%. → Against pace on a good or short-of-good length, 47.9% of his boundary runs go leg side. The average batter’s figure is 34.9%. The single most valuable fielder is forward square leg, saving 18.3% of his boundary runs. → Slog shots make up 20.1% of his runs, against 7.6% for an average batter. All of it came back in seconds, in plain cricket language. @taniiishh and I have spent two years in cricket analytics, building metrics and algorithms to read T20 better. One thing kept coming up: coaches and analysts come from different worlds, and good numbers often never reach the dressing room. So we built CricBit. A coach just asks, and the AI explains our metrics the way a coach would. It’s plug and play with any ball-by-ball data. And we’ve built one of the biggest collections of Indian state T20 league data out there (TNPL, the Maharaja Trophy, the UP T20, the DPL and many more). That’s cricket almost nobody has ball by ball, and it’s where the next auction steals are playing right now. The IPL auction is just over a month away. If you scout, analyse or coach a T20 side, our DMs are open. Send us a name and we’ll show you what we see.
3
14
4
62
8,164
I asked people which data companies influence research direction at the frontier: 16 votes: Surge (@echen) 13 votes: Proximal (@calvinchen, @MatternJustus) 8 votes: Preference Model (@jennifgcrl, @Ning_Catsnail) 7 votes: Datacurve (@serenaa_ge, @charleyslee) 6 votes each: - Mechanize (pre-acquisition) (@tamaybes, @EgeErdil2, @MatthewJBar) - Mercor (@BrendanFoody, @adarsh_exe, @suryamidha) 5 votes: Fleet (@nicoup, @andrewthezhou) 4 votes each: - AfterQuery (@CarlosGeorgescu, @spencermateega) - Prime Intellect (@vincentweisser, @johannes_hage) 3 votes: Latch (@AlfredoAndere, @kenbwork, @kylegiff) 2 votes each: - Abundant (@huyouare, @kehuangsf, @meji_abidoye) - Artificial Analysis (@_micah_h, @grmcameron) - Bespoke Labs (@madiator, @AlexGDimakis) - micro1 (@aliansarinik) - Terminal-Bench (@Mike_A_Merrill, @alexgshaw) 50 people voted. People couldn't pick a company they work at or founded. Disclosure: on this list, I'm a small investor in Abundant, AfterQuery, Arcanum, DLR, Fleet, Latch, Paper Instruments, Snorkel, and Vmax. I didn't vote. Continued:
13
18
4
287
45,890
Akshit retweeted
Hi! this is Rohan Anil, I develop training algorithms among other things - my work trains various frontier models. At CoreAuto, I was fascinated by the problem of training on compute islands, eventually in space training. I hope you all enjoy this write up!🪐🛰️ coreauto.com/hitchhikers-gui…
66
136
39
1,581
192,361
New paper: Benchmarks Are Surprisingly Compaction Friendly Agents commonly go past their context windows, and then "compact" to keep going. Pretty much all frontier agents do this, but no one has evaluated compaction on actual long horizon benchmarks We tested 5 such compaction strategies
4
14
3
97
7,780
Absolutely breathtaking view of the mighty Kosi River in Supaul, Bihar, with the Himalayas painting the horizon. @TourismBiharGov must look beyond the Buddhist circuit and develop these hidden gems to put Bihar on the tourism map.
82
702
108
4,813
303,927
New Anthropic research: How AI agents shape experts' judgment calls When experts delegate work to an agent, do they still make the key decisions? We tested this in a randomized trial with 50 professional data analysts.
336
560
434
4,343
828,855
Gemini 4 Carbon is our next frontier model. It’s state-of-the-art across cybersecurity and long-horizon capabilities.
750
1,389
1,100
14,527
1,733,829
My favorite genre of image, if you have more below please feel free to add
143
12,689
248
92,880
1,129,765
Replying to @MalleableTwig
Yes, sure thing! Here is a link to my slides. (The talks were recorded, but I am not sure whether/when they will make their way online.) docs.google.com/presentation…
2
5
1
35
8,874
Mt Kanchenjunga on a full moon night. Kurseong.
22
71
3
1,534
21,695
Akshit retweeted
A few bowhead whales born around 1815 might still be alive. They were in their 30s when they first saw a commercial whaling ship; in their 60s when they first heard a propellor; over 100 the first time they heard sonar. I wonder if they remember the silent ocean.
the reason moby-dick remains such a vital piece of human art is that it addresses what is probably one of humanity's greatest collective atrocities, the genocide of the whales, with an ambivalence it cannot itself fully articulate. melville is distraught, but he can't explain why
6
1,015
29
12,618
130,577