
Professor @nyuling and @NYUDataScience, research scientist @GoogleAI, inventor of the word "bertology"
NYC
Joined April 2009
- Tweets16.8K
- Following1K
- Followers20.6K
- Likes33.8K
Pinned Tweet
Quick break from jokes to share that I'm on leave from NYU this year and spending more of my time at Google, where I'm working to develop new ways to evaluate Gemini! I'm still based in NYC and checking in on my NYU group regularly to make sure their agents are not going rogue.
"If we don't do it, other people will do it first" is a pretty deflationary view of AI research: it's routine and predictable enough that everyone can do it, but the really good researchers can do it 10% faster than everyone else and win.
If the big AI safety foundations allocated at least some of their funds through an independent arm's length review panel (the NSF model), that would go a long way to promote intellectual diversity and accountability in the field.
This mechanism could literally be NSF! I know they've collaborated with external funders before (e.g. Amazon here amazon.science/academic-enga…), but not sure about the details.
3
383
someone at Anthropic could just write a reviewing skill and fix 95% of this
Human reviewers pull 8.4% of their criticisms from a paper's appendix.
Frontier LLM reviewers: 62%.
A new study had GPT-6 and Claude 5.5 models review ICLR 2027 submissions, then mapped each weakness to the section it came from. The AI critiques keep going past page 30 of the appendix.
They skew toward detail audits: more seeds, significance tests, confidence intervals. Rarely the core idea.
Same pattern via API, web chat or coding agent, which points at the models, not the interface.
arxiv.org/abs/2611.17811
#PeerReview #LLMs #AcademicTwitter
in nyc we say “stand on line” instead of “stand in line”, and “how many more sets do you got” instead of “could you please stop using the pec fly as a tiktok viewing station”
1
14
1,747
Carbon is a bit more straightforward, with fewer affectations, which is nice. But the main thing for me is it doesn't use nearly as many "unilateral conceptual pacts" as Opus - the thing where Opus uses inscrutable shorthands for a concept that was introduced into the conversation 18 hours ago. Here I often truly don't understand what it is trying to refer to.
Working with Opus 5.5 instead of Carbon today to change things up a little. The style is pretty unique, interesting that Anthropic chose to train Opus 5.5 on ICLR submissions!
2
43
3,794
Tal Linzen retweeted
Excited to release a new draft handbook chapter on Language Models and Experimental Syntax!
We review the last ~10 years of work using experimental syntax methods (minimal pairs, acceptability judgments, priming) to ask if LMs can do syntax
w/@qyao23,@kanishkamisra & @kmahowald
Tal Linzen retweeted
New blog post: I used Claude to audit the reproducibility of every journal paper I've been an author on in the last five years. Our commitment to openness meant that the code and data were available, but there were some significant issues. 🧵babieslearninglanguage.blogs…
New NIH call for proposals on human reasoning - one of the areas of focus is on developing computational models. Very cool that NIH is interested in this grants.nih.gov/funding/find-…
(happy to have had a small role in this call thanks to this workshop organized last year by a few forward looking NIH program officers! obssr.od.nih.gov/news-and-ev…)
5
838
Tal Linzen retweeted
I tell my students to instinctively distrust any "emergence" claims about LLMs (and read "emergence" as "training"). We don't need magical phrasing to appreciate the usefulness of these mega-trained models.. – at Holden Heights, FL
These midterms reflect a sea change in this nation's attitude towards my favorite issue: bike lanes. If you're not seeing this, you're not paying attention. By 2028 we'll have a protected crosstown bike lane in Manhattan. Maybe even two.
Officially* calling prop 3 in NYC has passed, which reduces procedural requirements and litigation risk for DOT street projects, mainly bike lanes
*74% scanners reporting != 74% votes reported, but given the margins it's not so important
15
1,322
Great move. Bayesian intervals are both more intuitive and more useful.
We’ve updated the Epoch Capabilities Index (ECI) to use a Bayesian methodology. This improves uncertainty estimates and lets us compare models more accurately, even when their individual ECI uncertainty intervals overlap.
Frontier model scores are largely unchanged: models stronger than GPT-4 move by less than 1 ECI point on average.
1
10
1,810
Thanks, Ben, for talking to us (and others) about some of the recent work on LLM introspection:
After over a century of studying introspection in humans and animals, researchers are beginning to explore the phenomenon in unlikely new subjects — large language models. These systems, which are based on mathematical structures called neural networks, excel at cognitive tasks that until recently seemed exclusive to humans. Can they introspect as well? @benbenbrubaker reports: quantamagazine.org/new-resea…
I really like this pitch. This is how we should hire senior AI researchers.
Applications are now open for the Anthropic Residency, a paid, fixed-term research program for academics, technical founders, and staff-level engineers who want to make AI safe and beneficial for everyone. No ML background is required.
Apply by November 22: alignment.anthropic.com/2026…
whenever a new blog post makes the rounds about how the field of interpretability or whatever has been revolutionized, it is prudent to wait for people you trust to read it and determine if it is in fact revolutionary. revolutions do happen but they're a little less frequent than social media might make it seem.
















