Yale NLP retweeted
⚛️ We’ve updated our physics benchmark paper and released code, audit data, and expert-verified solutions!
📄 Paper: arxiv.org/abs/2609.13009
💻 Code & data: github.com/Allliance/physics…
📰 Happy to see our work covered in New Scientist too!
newscientist.com/article/259…
New paper: How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
Frontier models have tackled Navier–Stokes.
People naturally wonder, how is their capability for physical sciences? According to today’s benchmarks, frontier models still struggle with physics (e.g. 47.3% with GPT5.6-sol on Humanity Last Exam - Physics component). But that picture did not match physicists' experience using these models.
So we did a rigorous analysis of current benchmarks and re-graded their supposed failures with Yale physicists.
We found that the problem was often the benchmark and not the model !!
When corrected, the models almost saturate all benchmarks, including everyone’s favourite, Humanity Last Exam (physics-part) and Critpt. These failures in turn make AA less reliable as a measure of model’s true capability.
For more details, results and analysis check out our preprint:
arxiv.org/abs/2609.13009
and blog post:
jsous.github.io/blogs/is-phy…
Check out our new survey on metacognition and LLM!
Metacognition is increasingly seen as a cornerstone of capable, trustworthy AI. Yet how to define, measure, or improve it in LLMs remains unclear. Our paper presents the first comprehensive overview of this emerging field!👇
📄 arxiv.org/pdf/2607.11881
🔗 github.com/yale-nlp/LLM-Meta…
Yale NLP retweeted
🔥 New paper alert!! Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? 🔥
We increasingly trust LLM judges to supervise deep research agents. But can we trust the judges themselves? Turns out: even frontier models score below 50% 😯
Details 🧵👇
Yale NLP retweeted
🔥 New paper alert!! Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs 🔥
Can a model that learns to 𝘫𝘶𝘥𝘨𝘦 𝘪𝘵𝘴𝘦𝘭𝘧 get even better at tasks—and learn to be honest about what it doesn't know? Turns out: yes🤯
Details🧵👇
Yale NLP retweeted
Also glad to share this was accepted as a ✨spotlight✨to the Trustworthy AI4GOOD Workshop @ #ICML 2026!
Looking forward to seeing you today!!
When multimodal AI meets real-world expertise, reasoning gets harder, deeper, and much more exciting.
Join us at KnowledgeMR @ #CVPR2026 to push this frontier forward!
🗓️Thu June 4 | 8am | Room 704/706
Speakers: @thoma_gu @huang_biwei @pliang279 @MengdiWang10 @xwang_lk
Yale NLP retweeted
🔥 Excited to share my new preprint: Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence? 🔥
When an LLM says "I think" or "probably," does it actually mean something consistent internally? The answer: not really 😬
Check out details in 🧵(1/n):
Check out our work MetaFaith on LLM faithfulness in uncertainty expression and talk with @pybeebee at #EMNLP2025 this week!!
Traveling to #EMNLP2025 to present our work “MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs” this week! 🇨🇳
Come by & let's chat about LLM faithfulness / uncertainty / calibration!
📍Poster Session 7 @ Hall C3
🗓Fri, Nov 7 @ 2-3:30p
🔗tinyurl.com/metafaith
Yale NLP retweeted
📢 Our recent work - NLPBench: Evaluating Large Language Models on Solving NLP Problems
arxiv.org/pdf/2309.15630.pdf
🤔 Could LLMs answer college-level NLP questions like a human?
We introduce NLPBench, which consists of 378 college-level NLP questions spanning various NLP topics.
Yale NLP retweeted
🎉@RexYing0923 AosongFeng @dragomir_radev Our recent work accepted at #ACL2023, a hierarchical model for long document classification. We also standardized the benchmark with six existing ones.
paper link: arxiv.org/pdf/2305.03319.pdf
code link: github.com/IreneZihuiLi/HiPo…
Yale NLP retweeted
Lastly, I want to thank everyone that I’ve shared a meal, a coffee/drink or even just a conversation. I’m like 96.2% sure that I’m the only one from Yale that’s attending in person this year. Thanks for keeping me company and making me feel I’m not alone. 4/n,n=4 – at Rwanda
Yale NLP retweeted
Finding 3. The LM "curse of multilinguality" which causes per-language performance to drop as they cover more languages is minimal and inconsistent!
They conclude: some of the literature on the curse of multilinguality may need to be revisited using more diverse benchmarks.
4/5
Yale NLP retweeted
Background: The Palm paper (arxiv.org/pdf/2204.02311.pdf) found parallel attention+MLP to be 15% faster but degrades quality at models with <8B parameters.
3/5
Yale NLP retweeted
These findings from the Pythia paper (arxiv.org/pdf/2304.01373.pdf) by @BlancheMinerva @haileysch__ et al., are fascinating:👇
Finding 1- Deduplication of training data shows no clear benefit, contrary to literature.
1/5 🧵
Yale NLP retweeted
Salesforce AI #ACL2023 Papers!
My Highlights
* Socratic Pretraining arxiv.org/pdf/2212.10449.pdf
* Robust Summarization Evaluation arxiv.org/pdf/2212.07981.pdf
* Constractive Parameter Ensembling arxiv.org/abs/2110.07166
* Did you read the Instruction?
* SWIPE: Document Simplification
Check out our #ACL2023 Accepted Papers! Congrats to all the authors! @silviocinguetta @CaimingXiong @alexfabbri4 @JiachengNLP @memray0 @JotyShafiq @jasonwu0731 @iam_wkr @PhilippeLaban @jesse_vig @yingbozhou_ai @semih__yavuz
Yale NLP retweeted
Our LongEval paper just received an outstanding paper award at #EACL2023 🏆
LongEval provides a way to improve inter annotator agreement while minimizing annotator workload in challenging task of long form summarization
Great work by @kalpeshk2011 and the team! Details below👇
Happy to share that LongEval received an outstanding paper award 🏆 at #EACL2023! Thanks to @eaclmeeting and our reviewers for the support!
Interested in improving human evaluation of long text? See our paper, code and thread 👇
arxiv.org/abs/2301.13298
github.com/martiansideofthem…
Yale NLP retweeted
Ah, forgot to mention the most important things😅: link to paper (arxiv.org/abs/2205.14318); code is also available here (github.com/microsoft/TraceCo…) 8/7
