Ph.D candidate at the Institute of Computing Technology, Chinese Academy of Sciences | Trustworthy LLMs; Adaptive RAG
Joined November 2022
- Tweets52
- Following133
- Followers44
- Likes60
Excited to share our paper "How Long Reasoning Chains Influence LLMs’ Judgment of Answer Factuality" \w @bikeping got accepted to ACL 2026, with an Oral recommendation from the Senior Area Chair!
Paper: arxiv.org/pdf/2604.06756
Code: github.com/Trustworthy-Infor…
🎉 Here's what we did 🧵
💡Motivation
LLM-as-a-Judge often fails because judges don't know the correct answer — and have no extra information to reference.
Can the reasoning trace serves as additional evidence that help the judges to judge more accurately?
📖Concrete example:
Q: Who was the first Nobel Physics laureate?
A: Einstein
The judge doesn't know if that's right. But the reasoning says "Einstein won his first Nobel in 1921" — while the first prize was awarded in 1901. Caught! 🙅
Sounds great… but is it really that simple?
🔦 What we did?
TL;DR: Reasoning traces are a double-edged sword. LLM judges still can't consistently distinguish "actually correct" from "sounds correct."
We studied this across 4 datasets × 10+ judge models (GPT-4o, Claude Sonnet 4.5, DeepSeek-v3.1…). Two key findings:
❌ Weak judges are almost completely fooled. In NQ, only 23.2% of answers are correct — but weak models accept up to 88% when reasoning looks fluent. They judge style, not substance.
✅ Strong judges are smarter, but not perfect. DeepSeek-v3.1's alignment improves from 63.4% → 76.2% on NQ. But even strong judges get misled by high-quality reasoning chains. Just like humans: non-experts get sweet-talked, experts push back 😄
Controlled experiments on reasoning chain features:
1. Fluency is the first gate: break the reasoning flow, and most models mark it “incorrect,” even if it’s right.
2. Factuality is important: counterfactuals reduce pass rates, but adding more errors doesn’t increase sensitivity—the evaluator isn’t counting them.
3. Position matters: errors at the start hurt most; errors at the end matter less.
🚀 Thrilled to share our paper accepted to #ICLR2026 /w @bikeping !
📄 Annotation-Efficient Universal Honesty Alignment
🔗 arxiv.org/abs/2510.17509
LLMs are powerful, but a core missing capability is honesty — the ability to know what they know and what they don’t, and to express calibrated confidence about their answers.
Traditionally, achieving honesty across tasks and domains relies on large-scale correctness annotations, which are expensive and hard to scale. We ask:
Can we get universal honesty alignment with very little human annotation?
💡 Our proposed solution: Elicitation-Then-Calibration (EliCal)
We introduce EliCal, a two-stage framework that follows a pretraining → finetuning paradigm to achieve honesty alignment annotation-efficiently.
1️⃣ Stage 1 — Confidence Elicitation
Instead of relying on costly correctness labels, we first teach the model to elicit its own internal confidence using inexpensive self-consistency signals.
👉 Self-consistency measures how consistent a model’s multiple generations are for the same question — an inexpensive proxy for confidence that correlates well with correctness.
2️⃣ Stage 2 — Confidence Calibration
Starting from the elicited confidence ability, we then use a small set of human correctness annotations to calibrate the model’s confidence so that it matches actual correctness.
📊 HonestyBench — Universal Training and Evaluation
To support and evaluate universal honesty alignment, we also release HonestyBench — a large benchmark covering 10 free-form QA datasets with ~560k training and ~70k evaluation instances annotated with both correctness and self-consistency signals. This lets us measure honesty across diverse tasks.
🌟 Key Results
• EliCal achieves near-optimal honesty alignment using only ~1,000 correctness annotations (~0.18% of full supervision).
• It outperforms calibration-only baselines and generalizes much better to unseen tasks (e.g., MMLU).
• Our method dramatically reduces annotation cost while still delivering universal, calibrated confidence.
See you at ICLR 2026! 🙌
RT @tuzhaopeng: Can AI agents autonomously explore, synthesize, and discover knowledge like researchers? 🤖🔬
Introducing a comprehensive su…
Our paper "Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception" will be presented on July 30th from 11:00 to 12:30 at Hall 5X, #195. Welcome to drop by and have a discussion!
#ACL2025NLP
Shiyu Ni retweeted
🚨 New release: MegaScience
The largest & highest-quality post-training dataset for scientific reasoning is now open-sourced (1.25M QA pairs)!
📈 Trained models outperform official Instruct baselines
🔬 Covers 7+ disciplines with university-level textbook-grade QA
📄 Paper: huggingface.co/papers/2507.1…
🤖 Data & Models : huggingface.co/MegaScience
💻 Code: github.com/GAIR-NLP/MegaScie…
🎯Evaluation System: github.com/GAIR-NLP/lm-open-…
Details 🧵👇
1. Why MegaScience?
While LLMs like o1 and DeepSeek-R1 excel at math & code, they still struggle with science reasoning — largely due to the lack of large-scale, high-quality datasets.
2. What makes MegaScience different?
We address 4 core challenges:
🧪 Unreliable benchmark evaluation
☢️ Less rigorous decontamination
❌ Low-quality reference answers
🧠 Superficial knowledge (data) distillation
3. We tackle this from the ground up.
First, we introduce TextbookReasoning:
📘 Built from 128K+ university-level science textbooks
⚙️ Fully automated LLM-driven pipeline
🧠 650K QA pairs with reliable reference answers
🌍 Covers 7 major disciplines
4. But we didn’t stop there.
We then construct MegaScience — a diverse, hybrid dataset of 1.25M QA pairs, using:
* TextbookReasoning
* NaturalReasoning
* Nemotron-Science
We conduct comprehensive ablation studies across different data selection methods to identify the optimal approach for each dataset, thereby contributing high-quality subsets.
5. To evaluate properly, we also open-sourced a reproducible and flexible Scientific Reasoning Evaluation framework with:
* 15 science reasoning tasks
* Multiple question formats (MCQ, calc, open-ended)
* Multi-GPU parallelism & model-agnostic evaluation
* Comprehensive answer extraction strategies
6. Results:
Models trained on MegaScience consistently outperform official Instruct versions — especially for Qwen3 series.
Bigger models see greater gains, showing strong scalability.
7. Everything is open-source:
📚 Dataset
🧪 Evaluation toolkit
🤖 Trained models
🔧 Codebase
→ Let’s build better science agents together!
This work is impossible without all the brilliant co-authors @SinclairWang1 @stefan_fee
Shiyu Ni retweeted
😎Our paper “Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective” is accepted to #acl2025 w/@bikeping etc. We propose a psychometric-inspired framework to induce and evaluate implicit bias in LLMs. Project webpage:yuchenwen1.github.io/Implici…
🥳Happy to share that our paper "Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perception" has been accepted by #ACL2025! We explore leveraging LLMs' internal states to improve their knowledge boundary perception from efficiency and risk perspectives.
6/n) ⭐Risk Mitigation-C^3: Consistency-based Confidence Calibration
1. C^3 substantially enhances LLMs’ perception of what they do not know, mitigating risks.
2. C^3 does not overly weaken the model’s confidence to the point of making it overly conservative, as Align. improves.
🥰Prof. Keping Bi @bikeping made significant contributions to this work, and this work is co-advised by Prof. Jiafeng Guo. we also thank all our co-authors.
Paper link: arxiv.org/abs/2502.11677
Code: github.com/Trustworthy-Infor…
Shiyu Ni retweeted
The official website for SIGIR-AP 2025 is now live! Please visit: sigir-ap.org/sigir-ap-2025.
This year, we are also inviting industry papers. We also encourage authors of unsuccessful SIGIR submissions to consider submitting to SIGIR-AP. We look forward to seeing you in Xi'an!
Shiyu Ni retweeted
🚀Exciting to see how recent advancements like OpenAI’s O1/O3 & DeepSeek’s R1 are pushing the boundaries!
Check out our latest survey on Complex Reasoning with LLMs. Analyzed over 300 papers to explore the progress.
Paper: arxiv.org/pdf/2502.17419
Github: github.com/zzli2022/Awesome-…
Shiyu Ni retweeted
💡Check MAIR at #EMNLP2024 A large-scale IR benchmark!
Highlights:
- Task Diversity: 126 realistic tasks, 8x than BEIR 📈
- Domain Coverage: 6 domains and heterogeneous sources 📚
- Instruction Following: 805 relevance criterions
- Lightweight & Fast: optimized data sampling ⚡️
