How does On-Policy Distillation (OPD) actually work in LLM post-training? Our new paper shifts the focus from scaling the teacher to diagnosing the training signal.
🎯 Role
OPD acts as an exploration catalyst: it steers students toward correct reasoning paths via dense token-level signals, but does not raise the student's underlying capability ceiling.
⚠️ Pathologies
Relying on token-level rewards exposes two failure modes:
• Student–Teacher Mismatch — distribution gaps turn "guidance" into misleading noise
• Length Exploitation — students game rewards by padding / truncating responses
🛠️ Regulations
Two lightweight, near-zero-overhead in-loop fixes, advantage clipping and log-scale compression, suppress extreme outliers and keep the signal faithful.
💡 Takeaway — Stronger ≠ Better
With signal regulation, a 4B teacher outperforms baselines distilled from a 30B one. In OPD, it's signal quality, not teacher scale, that governs successful exploration.
📄 Paper: huggingface.co/papers/2607.1…

