Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding
arXiv cs.CL · 17 Oct 2026, 03:41 AEST
A new study reveals that relaxed verification in speculative decoding leads to longer completions despite accepting more tokens per round. This length inflation phenomenon suggests that accepted tokens alone are not a reliable measure of decoding speed.
Key facts
- Speculative decoding typically characterizes speedup by draft tokens accepted per verification round.
- We identify a length inflation phenomenon: relaxed verification lengthens completions, primarily by extending reasoning traces, largely offsetting apparent gains in accepted tokens per round.
- At each rule's most relaxed evaluated setting, relaxed verification on average accepts $1.34\times$ as many tokens per round but produces $1.54\times$ longer completions.
- Overall, accepted tokens per round alone is an unreliable measure of decoding speed under relaxed verification.
Read aloud
A new study reveals that relaxed verification in speculative decoding leads to longer completions despite accepting more tokens per round. This length inflation phenomenon suggests that accepted tokens alone are not a reliable measure of decoding speed. Speculative decoding typically characterizes speedup by draft tokens accepted per verification round. We identify a length inflation phenomenon: relaxed verification lengthens completions, primarily by extending reasoning traces, largely offsetting apparent gains in accepted tokens per round. At each rule's most relaxed evaluated setting, relaxed verification on average accepts one dollars and thirty four cents\times$ as many tokens per round but produces one dollars and fifty four cents\times$ longer completions. Overall, accepted tokens per round alone is an unreliable measure of decoding speed under relaxed verification.
Sources in this cluster
- Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding
arXiv cs.CL · 17 Oct 2026, 03:41 AEST