Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
arXiv cs.AI · 15 Oct 2026, 03:54 AEST
A new method called CriticOPD has been proposed to enhance on-policy distillation by addressing the issue of teacher recoverability. This approach allows a student model to learn from critiques of its errors, leading to improved performance on mathematical reasoning tasks.
Key facts
- On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes.
- We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them.
- The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error.
- On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Read aloud
A new method called CriticOPD has been proposed to enhance on-policy distillation by addressing the issue of teacher recoverability. This approach allows a student model to learn from critiques of its errors, leading to improved performance on mathematical reasoning tasks. On-policy distillation OPD trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. The teacher critiques an incorrect rollout, allowing the student to rewrite its solution following the error. On two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD improves OPD, with the largest gains on competition problems.
Sources in this cluster
- Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
arXiv cs.AI · 15 Oct 2026, 03:54 AEST