Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards
arXiv cs.AI · 14 Oct 2026, 03:58 AEST
A new study explores the dynamics of reinforcement learning with verifiable rewards, focusing on the impact of larger rollout batches. The findings suggest that while larger batches may seem efficient, they can lead to slower learning and increased risk of failure.
Key facts
- Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling.
- Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades.
- Controlled GRPO experiments on Qwen2.5-Math-1.5B across a $16\times$ batch range support both predictions.
- Ratio clipping delays failure, especially for larger batches, but the progress gap persists without it.
Read aloud
A new study explores the dynamics of reinforcement learning with verifiable rewards, focusing on the impact of larger rollout batches. The findings suggest that while larger batches may seem efficient, they can lead to slower learning and increased risk of failure. Reinforcement learning with verifiable rewards RLVR relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-rate-dependent critical batch beyond which sample efficiency degrades. Controlled GRPO experiments on Qwen2.five-Math-one.5B across a sixteen dollars\times$ batch range support both predictions. Ratio clipping delays failure, especially for larger batches, but the progress gap persists without it.
Sources in this cluster
- Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards
arXiv cs.AI · 14 Oct 2026, 03:58 AEST