NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding
arXiv cs.AI · 24 Oct 2026, 02:47 AEST
NgramRL is a new method designed to enhance reinforcement learning post-training for large language models by improving rollout generation efficiency. It utilizes a per-prompt n-gram drafter that leverages recent rollout history without requiring additional model training or GPU resources.
Key facts
- Reinforcement learning (RL) post-training improves the reasoning capabilities of large language models, but its cost is dominated by autoregressive rollout generation, which can exceed 70% of training time.
- We propose NgramRL, a per-prompt n-gram drafter constructed from bounded recent rollout history.
- NgramRL requires no auxiliary model, drafter training, or persistent GPU-resident drafter state, while target-model verification preserves the rollout distribution.
- Across two model families and three reasoning and structured-generation workloads, NgramRL commits 3.60-7.77 tokens per target-model forward in the warm-cache regime, up to 1.99x more than SuffixDecoding and 7.0x more than prompt-lookup decoding.
Read aloud
NgramRL is a new method designed to enhance reinforcement learning post-training for large language models by improving rollout generation efficiency. It utilizes a per-prompt n-gram drafter that leverages recent rollout history without requiring additional model training or G P U resources. Reinforcement learning RL post-training improves the reasoning capabilities of large language models, but its cost is dominated by autoregressive rollout generation, which can exceed seventy percent of training time. We propose NgramRL, a per-prompt n-gram drafter constructed from bounded recent rollout history. NgramRL requires no auxiliary model, drafter training, or persistent G P U-resident drafter state, while target-model verification preserves the rollout distribution. Across two model families and three reasoning and structured-generation workloads, NgramRL commits three point six zero-seven point seven seven tokens per target-model forward in the warm-cache regime, up to one.99x more than SuffixDecoding and seven.0x more than prompt-lookup decoding.
Sources in this cluster
- NgramRL: Accelerating RL Post-Training for Structured Generation via Rollout-Aware N-gram Speculative Decoding
arXiv cs.AI · 24 Oct 2026, 02:47 AEST