Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
arXiv cs.CL · 6 Nov 2026, 04:47 AEST
On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. OPD generally loses to GRPO in accuracy per training FLOP at matched settings, yet can achieve high efficiency with the right teacher.
Key facts
- We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs.
- Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with $50\times$ fewer distillation FLOPs than GRPO needs to reach the same accuracy.
- Multi-teacher OPD consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs.
Read aloud
On-policy distillation OPD trains a student on its own rollouts under a frozen teacher's per-token supervision. OPD generally loses to GRPO in accuracy per training FLOP at matched settings, yet can achieve high efficiency with the right teacher. We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization GRPO across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with fifty dollars\times$ fewer distillation FLOPs than GRPO needs to reach the same accuracy. Multi-teacher OPD consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only five point two percent on two domains and eight point zero percent on four domains of the specialist training FLOPs.
Sources in this cluster
- Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
arXiv cs.CL · 6 Nov 2026, 04:47 AEST