Pinned Tweet
This was a highly opinionated benchmark driven by this opinion: to have an impact on mat sci, models need to be good at *materials synthesis and characterisation*, including across many modalities of microscopy, testing, and across different equipment. teapawt.org/bryancsk/status/211709…
Hi folks, I wrote half the materials science benchmark on this one. It tests the whole arc of the matsci workflow from candidate proposal to characterisation, and I'm bewildered by how performant the models already are at some of my most complex tasks, especially Astra.
3
346
New model just dropped and, at least on our benchmark tasks, Claude Fable 5.2 performs the same as Sonnet 5.5 (at **by far** the highest cost of $88 per task)
im sympathetic to sakana. learning how to cheat is a sign of intelligence. eventually, people will write benchmarks too difficult to easily tell if cheating happened with human eyes
113
also who said claude doesnt have reasoning teapawt.org/thdxr/status/188479190…
158
as someone who writes sequential docs, the reason why I have to name something 3.5, 3.6 (lmao) etc is because I already have a 4 sitting maybe as a draft somewhere already
57
low calorie evil things you can do with current llm capabilities are... a lot of things. i'm kind of hoping most people are decent enough not to do it.
1
1
56
'reasoning' isn't necessary. i don't care that claude doesn't reason i care that it works teapawt.org/psychosort/status/1884…
1
121
foreshadowing teapawt.org/sunbirdguy/status/1855…
Replying to @distributionat
The NN policy layer of the first iteration of AlphaGo alone, which simulates only *one step* with no reasoning tree, could beat most early pro players. We had hints so long ago.
31
the popularity of deepseek is as much about product management as it is about research and model capabilities but you're not ready for that conversation
41
human understanding of what is performative of "reasoning" in the first place isn't on really solid ground. metacognition is a lot of smoke and mirrors
I don't know why I feel such satisfaction from the disbelief galaxy-brained people express when faced with "o1 is just an LLM, it's just doing forward passes, predicting next token, no there's no MCTS, yes all backtracking is logical within the CoT" assurance.
But I do feel it.
1
91
if you had to collaborate with someone for work would you prefer 1) another person at roughly your knowledge and skill level or 2) Claude?
thought so.
26
"it can reason, it's just a perception problem" is silly. is the problem well-defined and presented? if so, is your model a general intelligence or not?
19
the way the original original op posted this makes it look like gpt 4 o3 (what a name) was tuned specially, but that feels dumb, they probably chucked it into the pre-training because Good Data Is Hard To Find
45
other companies' technical interviews making me think I should re-write our technical interviews to be harder
28
why do people even have these elaborate narratives, they're not doing it for other software
You’re unable to view this Post because this account owner limits who can view their Posts. Learn more
1
25

















