Adaptive Multi-Token Prediction: How far should we go?
arXiv cs.AI · 21 Oct 2026, 03:56 AEST
Adaptive Multi-Token Prediction (AMTP) enhances language-model decoding by optimizing the token drafting process. It achieves significant speed improvements while maintaining accuracy across various settings.
Key facts
- Multi-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass.
- We therefore formulate horizon selection as an online control problem, using verification feedback to learn how far the draft can be trusted in each context, and introduce Adaptive Multi-Token Prediction (AMTP): a lightweight controller that requires no retraining and leaves the exact verifier unchanged.
- Across five draft-target pairings built on Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, evaluated on AIME 2025 and GSM8K, AMTP decodes up to 4.1x faster than standard decoding and matches or outperforms the fixed window in 8 of 10 settings, with throughput gains of up to 10%.
Read aloud
Adaptive Multi-Token Prediction AMTP enhances language-model decoding by optimizing the token drafting process. It achieves significant speed improvements while maintaining accuracy across various settings. Multi-token prediction MTP can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass. We therefore formulate horizon selection as an online control problem, using verification feedback to learn how far the draft can be trusted in each context, and introduce Adaptive Multi-Token Prediction AMTP: a lightweight controller that requires no retraining and leaves the exact verifier unchanged. Across five draft-target pairings built on Llama-three point one-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-two thousand five hundred seven-MTP, evaluated on AIME twenty twenty five and GSM8K, AMTP decodes up to four.1x faster than standard decoding and matches or outperforms the fixed window in eight of ten settings, with throughput gains of up to ten percent.
Sources in this cluster
- Adaptive Multi-Token Prediction: How far should we go?
arXiv cs.AI · 21 Oct 2026, 03:56 AEST