Tianshi Zheng (@Tianshi_0218) header
Tianshi Zheng (@Tianshi_0218) profile picture

Ph.D. candidate at HKUST. LLM, Agent, Discovery #NLProc

Clear Water Bay, Hong Kong
Joined May 2023
But here's our most surprising finding: The Paradox of Code Assistance. 😮 Giving models a code interpreter can paradoxically HURT the performance of the most capable models. Why? It's a classic exploration vs. exploitation trade-off. 🌱 Weaker models use the tool for basic math, freeing them up to explore new hypotheses. 🎯 Stronger models over-rely on it for quick function-fitting, getting stuck in "good enough" local optima and abandoning the search for the true law. (6/7)
1
40
This fragility is starkly clear when we add noise. Real-world data is messy, so we tested for robustness. The result? Performance craters. 📉 Even a tiny 0.01% of observational noise caused symbolic accuracy to drop by ~15%. This shows that current models are extremely sensitive and struggle with the imperfections of real-world data. (5/7)
1
25
So, how did today's top LLMs (incl. GPT-5, Gemini-2.5-Pro, DeepSeek-R1) perform? The results show a clear but fragile capability. 🧊 🔹 Frontier models ace simple tasks (>80% accuracy). 🔹 But performance 📉 plummets as complexity increases. In our challenging environments, GPT-5 hit just 29.9% accuracy, while others fell below 5%. Robust discovery is still a major hurdle. (4/7)
1
51
Science isn't about fitting equations to a static spreadsheet 📊. It’s about active, hands-on exploration. That's why NewtonBench isn't a dataset. It's an interactive lab. 🔬 Agents must design experiments, probe complex systems with confounding variables, and interpret feedback to uncover hidden laws. It’s a true test of the scientific method. (3/7)
1
1
54
Current benchmarks are stuck in a trilemma: they must sacrifice scientific relevance, scalability, or resistance to memorization. Our solution? Metaphysical Shifts. 🌌 We systematically alter canonical laws to create novel, physically plausible "parallel universes." This forces models to reason from first principles, not just recall what they already know. No memorization. Just pure reasoning. 🧠 (2/7)
1
66
🤯 Can LLM rediscover Newton’s laws? Imagine an LLM agent perched in the apple tree in 1666. Could it independently uncover the law of gravitation? ✨ Introducing NewtonBench: a benchmark for genuine scientific discovery—by launching “parallel universes” for LLMs to explore👇 Paper: arxiv.org/pdf/2510.07172 Code: github.com/HKUST-KnowComp/Ne… (1/7)
1
3
1
8
858
We chart the paradigm shift as LLMs evolve from tools to autonomous agents in science. Introducing a foundational 3-level taxonomy (Tool, Analyst, Scientist) to map their escalating autonomy & capabilities. 🤖🔬
1
198
🚀 Our new survey "From Automation to Autonomy: LLMs in Scientific Discovery" is live! 📜 Paper: arxiv.org/pdf/2505.13259 ⭐ Awesome Github Repo: github.com/HKUST-KnowComp/Aw… #LLMs #ScientificDiscovery #AI #NLP #Agent #AIforScience @HKUSTKnowComp @hkustNLP (cont'd)👇
1
5
7
812