Agentic-OPD: Recovering Forgotten Agent Capabilities via On-Policy Distillation for Agentic Tasks
arXiv cs.AI · 10 Nov 2026, 04:07 AEST
Agentic-OPD introduces a method for recovering lost capabilities in reinforcement learning for agentic tasks. This approach utilizes a pool of discarded checkpoints to enhance the performance of language models.
Key facts
- Reinforcement learning is now the mainstream paradigm for training LLMs on agentic tasks, yet standard practice keeps only the best-validation checkpoint and discards the rest.
- We show that this is lossy: in GRPO training of LLM agents, the best-validation checkpoint fails on problems that earlier checkpoints from the same run had already solved.
- The discarded checkpoints thus form a teacher pool obtained at no additional training cost, which on-policy distillation (OPD) can mine to recover the lost coverage.
- Across five agentic task domains and 24 benchmarks and task subsets, Agentic-OPD improves over its initialization checkpoint by up to 7.4 points.
Read aloud
Agentic-OPD introduces a method for recovering lost capabilities in reinforcement learning for agentic tasks. This approach utilizes a pool of discarded checkpoints to enhance the performance of language models. Reinforcement learning is now the mainstream paradigm for training L L M s on agentic tasks, yet standard practice keeps only the best-validation checkpoint and discards the rest. We show that this is lossy: in GRPO training of L L M agents, the best-validation checkpoint fails on problems that earlier checkpoints from the same run had already solved. The discarded checkpoints thus form a teacher pool obtained at no additional training cost, which on-policy distillation OPD can mine to recover the lost coverage. Across five agentic task domains and twenty four benchmarks and task subsets, Agentic-OPD improves over its initialization checkpoint by up to seven point four points.
Sources in this cluster
- Agentic-OPD: Recovering Forgotten Agent Capabilities via On-Policy Distillation for Agentic Tasks
arXiv cs.AI · 10 Nov 2026, 04:07 AEST