Rui WANG (@RayAlle97028330) profile picture

PhD student at CUHK.

Joined November 2021
Photo by Rui WANG (@RayAlle97028330)
How does On-Policy Distillation (OPD) actually work in LLM post-training? Our new paper shifts the focus from scaling the teacher to diagnosing the training signal. 🎯 Role OPD acts as an exploration catalyst: it steers students toward correct reasoning paths via dense token-level signals, but does not raise the student's underlying capability ceiling. ⚠️ Pathologies Relying on token-level rewards exposes two failure modes: • Student–Teacher Mismatch — distribution gaps turn "guidance" into misleading noise • Length Exploitation — students game rewards by padding / truncating responses 🛠️ Regulations Two lightweight, near-zero-overhead in-loop fixes, advantage clipping and log-scale compression, suppress extreme outliers and keep the signal faithful. 💡 Takeaway — Stronger ≠ Better With signal regulation, a 4B teacher outperforms baselines distilled from a 30B one. In OPD, it's signal quality, not teacher scale, that governs successful exploration. 📄 Paper: huggingface.co/papers/2607.1…
1
4
1,558