We studied Claude, GPT, Gemini, GLM, Kimi, DeepSeek and LongCat on 36 AI R&D tasks, looking beyond final scores to how they research.
Takeaway: agents act more like engineering optimizers than autonomous researchers. Only 3 of 252 best solutions were novel.
arxiv.org/abs/2608.13417
#SelfEvolving #AutoResearch #AIResearch
An agent runs experiments for hours and improves a score. But did it actually do research, or just optimize?
We evaluate 7 frontier models on 36 long-horizon tasks, looking beyond outcomes into process, experience, novelty, and harness effects.
arxiv.org/abs/2608.13417
AI may not be reliable enough to review papers, but it is increasingly good at checking code.
Code agents could make academic artifacts genuinely testable: does the code run, do results reproduce, and does the implementation match the method?
That could be a real boost for #reproducibility.
1
79
🚀 Our paper “Fine-Tuning Done Right in Model Editing” is accepted to @iclr_conf 2026! Huge congrats to @10k_miles_yang
Fine-tuning is NOT weak for model editing. The pipeline was suboptimal. Implementation matters more than we thought.
We show that standard breadth-first fine-tuning + localized tuning beats SOTA, scales to 100K edits and 72B models.
📄 Paper: arxiv.org/abs/2509.22072
💻 Code: github.com/WanliYoung/FT4Edi…
#ModelEditing #KnowledgeEditing #finetuning #ICLR2026
This is @10k_miles_yang’s 4th paper so far, in his 2nd year of PhD. Really impressive progress.
89
📢 Excited to present our EMNLP 2025 paper “Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs”!
🗓 Wed, Nov 5 · 16:30–18:00
📍 Hall C
#EMNLP2025 #LLMs #AISafety #Hallucination #Uncertainty
2
3
487
🎉 Our paper "AuditAgent: LLM Agent for Risks Auditing in Recommender Systems" has been accepted to the AAAI 2026 Demo Track!
AuditAgent uses LLM-powered GUI agents to audit platforms for filter bubbles, unfairness, and data misuse.
#AAAI2026 #LLM #guiagent #algorithm_auditing #GUIAgent
THX for sharing our work!😀
Another great @GoogleDeepMind paper.
Shows how to speed up LLM agents while cutting cost and keeping answers unchanged.
30% lower total cost and 60% less wasted cost at comparable acceleration.
Agents plan step by step, so each call waits for the previous one, which drags latency.
Speculative planning fixes that by having a cheap draft agent guess next steps while a stronger agent checks them in parallel.
Fixed guess lengths backfire, small guesses barely help, big guesses waste tokens when a check disagrees.
Dynamic Speculative Planning learns how far to guess, then stops early to avoid wasted calls.
A tiny online predictor learns how many steps will be right using reinforcement learning.
1 knob lets teams bias for speed or cost, either by skewing training or adding a small offset.
If a guess is wrong, extra threads stop and execution resumes from the verified step.
Across OpenAGI and TravelPlanner, the dynamic policy matches the fastest fixed policy while spending fewer tokens
The result is clear, faster responses, lower bills, and 0 loss in task quality.
105
If you sample an LLM 15× and get the same answer, should you trust it? 🚨Not always!
⚠️ Consistency ≠ correctness.
🚫Scaling doesn’t help: self-consistent errors stay stable—or even increase—in larger models.
Our #EMNLP25 main paper (Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs) formalizes self-consistent errors, shows mainstream detectors miss them, and introduces a simple cross-model probe that consistently improves AUROC across datasets and model families.
preprint: arxiv.org/abs/2505.17656
#Hallucination #LLM_uncertainty #HallucinationDetection #LLM
4
354
📍 Find us at ACL 2025 – Hall 5X, Poster #83
🌐 More details & resources: yangwl.site/revisit-editing-…
See you there!
#ACL2025 #ModelEditing #KnowledgeEditing
🛑 Stop using teacher forcing to evaluate model editing!
Our ACL 2025 poster shows why past evaluations mislead progress & how to test editing in the wild.
📍 July 30, 11:00 AM – come chat!
#ModelEditing #LLM #ACL2025NLP
4
304
🛑 Stop using teacher forcing to evaluate model editing!
Our ACL 2025 poster shows why past evaluations mislead progress & how to test editing in the wild.
📍 July 30, 11:00 AM – come chat!
#ModelEditing #LLM #ACL2025NLP
🎉 Excited to share that our work "The Mirage of Model Editing" has been accepted as a main conference paper at #ACL2025! Many thanks to my supervisor @fei__sun, our collaborators, and special thanks to @HuaWenyue31539 for insightful discussions!
Great post! Thanks for featuring the figure from our #ACL2025 paper “The Mirage of Model Editing.”arxiv.org/abs/2502.11177 Glad to see the community's attention to evaluation challenges in knowledge editing. #LLMs #ModelEditing #KnowledgeEditing
🚨 New Blog Drop!
🚀 "Reflection on Knowledge Editing: Charting the Next Steps" is live!
💡 Ever wondered why knowledge editing in LLMs still feels more like a lab experiment than a real-world solution? In this post, we dive deep into where the research is thriving — and where it's falling short. From foundational breakthroughs to the practical roadblocks no one’s talking about, we connect the dots and propose what’s needed to move forward. Join the conversation! #KnowledgeEditing #LLMs #AI #ModelEditing
📌 If you're working on LLMs, model updates, or mechanism interpretability, you don’t want to miss this.
👉 Read the full post: yyzcowtodd.cn/rethinkedit
Key insights from our analysis:
0⃣ Current evaluation metrics and benchmarks inadequately assess knowledge updates in LRMs, highlighting the need for more comprehensive evaluation frameworks.
1⃣ Scaling challenges persist, with significant memory and computational constraints limiting the practical application of editing methods for larger or quantized local models.
🎁 Resource Release: To support the research community, we release covariance matrices for Qwen2.5-32B & QwQ-32B models for the current locate-and-edit methods.
2⃣ We outline promising research directions for developing language models that can effectively learn, adapt, and evolve their knowledge base.
Huge thanks to the brilliant collaborators who made this deep dive into #ModelEditing possible! @uclanlp @CanyuChen3 @Jiachen_Gu @dsmall2apple1 @ManlingLi_ @VioletNPeng
1
2
371
Sun Fei retweeted
🎉 Excited to share that our work "The Mirage of Model Editing" has been accepted as a main conference paper at #ACL2025! Many thanks to my supervisor @fei__sun, our collaborators, and special thanks to @HuaWenyue31539 for insightful discussions!
Your evaluation of model editing might be wrong !! Great work! 🎉 @aclmeeting #ACL2025 (main) #ModelEditing
Sun Fei retweeted
🚀 #InductionBench Super difficult benchmark alert! Even o3-mini achieve 5% accuracy!
LLM reasoning benchmarks have long been focused on math, code, and knowledge domains. But we’ve actually missed a huge area -- inductive reasoning:
✨ from limited data, can you generate the simplest underlying function? ✨ a.k.a. the minimum-length description, which is also a white-box explanation.
🤔🤔It’s also what most data analysts and scientific researchers care about: given limited data, how do we discover the underlying principle in limited time?
✨📊🔑 Truth is always simple.
📐 Previously, there were indeed some inductive reasoning benchmarks (like concept learning, etc.), but the entire field of inductive reasoning actually has very strict mathematical definitions:
1⃣ what class of function are we dealing with
2⃣ is there a unique optimal solution
3⃣ how big is the search space
4⃣ how much data is needed
5⃣ and in how much time can we find the shortest description?
This is the central idea of my article:
In a strictly defined inductive function class, how capable are LLMs?
We chose the simplest function class for inductive reasoning: regular functions/transducers, and within the subregular hierarchy of regular transducers, we picked the subregular classes with the lowest complexity: ISL, L-OSL, and R-OSL. According to existing research and proofs, these three non-comparable classes can yield a unique shortest description with polynomial data and polynomial computation time.
😱 I selected several very simple settings as target functions in these classes, and o3-mini’s performance was only 5% success rate.
🚧❌In other words, current LLMs have a significant flaw in inductive summarization, and this flaw makes me feel that their possibilities in #AI4Sci are greatly reduced.
Moreover, few-shot learning doesn’t help much with this type of reasoning, and the more data there is, the harder it becomes (after ruling out context length issues).
🤖🔥🏆 Of course, given that these three function classes have already been thoroughly researched, to avoid people using existing algorithms to “cheat,” I created an exploration version of the benchmark.
From the definitions, this function class only looks slightly more complicated than those three function classes. But as of now, there’s no mature research, no proof, no algorithm for it. 🏆🏆So if an LLM can succeed on the exploration benchmark, that would truly qualify as superhuman!
arxiv：arxiv.org/pdf/2502.15823
GitHub：github.com/Wenyueh/inductive…
#LLM #InductiveReasoning #MachineLearning #DataAnalysis #Benchmark #RegularTransducer #ISL #LOSL #ROSL #Ai4Sci #HuggingFace #SuperhumanPotential
Sun Fei retweeted
Call for Papers: Workshop on Human-Centered Recommender Systems @TheWebConf 2025
Updated Submission Deadline: January 6, 2025 (AoE)
We're excited to invite researchers to submit papers for the Workshop on Human-Centered Recommender Systems (HCRS) at TheWebConf 2025, happening from April 28 to May 2 in Sydney, Australia.
As recommender systems become increasingly pervasive, it's crucial to address challenges related to user privacy, fairness, transparency, and ethics. This workshop will focus on advancing systems that align with human values, fostering inclusivity and accountability. We welcome submissions on topics like fairness, bias, diversity, privacy, HCI design, ethical implications, and more.
We look forward to engaging in meaningful discussions towards building more responsible, transparent, and user-centered recommendation technologies.
For submission details, visit: human-centeredrec.github.io
@TheWebConf
#TheWebConf25 #WWW25 #RecSys
Sun Fei retweeted
Looking forward to seeing your research on human-centered recommender systems! We're welcoming submissions on topics like robustness, privacy, fairness, and more. Let's meet at @TheWebConf 2025 in Sydney, Australia! Submit by December 18, 2024 (AoE). #TheWebConf
Call for Papers: Workshop on Human-Centered Recommender Systems at TheWebConf 2025
We invite you to submit your research and join us at the Workshop on Human-Centered Recommender Systems (HCRS) during TheWebConf 2025, from April 28 to May 2 in Sydney, Australia.
Recommender systems are integral to daily life but pose challenges like information bubbles, privacy issues, and fairness concerns. This workshop aims to explore the development of systems that prioritize human needs, values, and capabilities. Topics of interest include robustness, privacy, transparency, fairness and bias, diversity, ethics, accountability, human-computer interaction design, evaluation, and auditing.
Submission Deadline: December 18, 2024 (AoE)
Join us to discuss and advance towards more ethical, user-centric, and socially responsible recommender systems.
For more information, please visit our website: human-centeredrec.github.io/
@TheWebConf
#TheWebConf25 #WWW25 #RecSys