Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
arXiv cs.CL · 21 Oct 2026, 03:40 AEST
A new method called Rollout-to-Scorer Distillation (R2S) aims to improve agentic on-policy distillation by addressing the limitations of traditional scoring methods. R2S repairs the teacher's scoring process to enhance the training of student agents without relying on task rewards.
Key facts
- Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token.
- We attribute this to the myopia of next-token scoring.
- We propose Rollout-to-Scorer Distillation (R2S), which repairs the scorer before distillation by mining high-disagreement states from the untrained student.
- Across three environments, two teachers and four students, R2S outperforms three state-of-the-art agentic-OPD methods in all twelve teacher-student-environment cells.
Read aloud
A new method called Rollout-to-Scorer Distillation R2S aims to improve agentic on-policy distillation by addressing the limitations of traditional scoring methods. R2S repairs the teacher's scoring process to enhance the training of student agents without relying on task rewards. Agentic on-policy distillation OPD trains a student agent on its own trajectories while a teacher scores every token. We attribute this to the myopia of next-token scoring. We propose Rollout-to-Scorer Distillation R2S, which repairs the scorer before distillation by mining high-disagreement states from the untrained student. Across three environments, two teachers and four students, R2S outperforms three state-of-the-art agentic-OPD methods in all twelve teacher-student-environment cells.
Sources in this cluster
- Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
arXiv cs.CL · 21 Oct 2026, 03:40 AEST