AI TrackerFrontier desk · AEST
Back to news
normalaitech

RangeSWA: Training-Free Length Extrapolation for Continuous Diffusion Language Models

arXiv cs.CL · 31 Oct 2026, 03:35 AEST

RangeSWA is a training-free method designed to improve length extrapolation in continuous diffusion language models. It addresses issues like text repetition and semantic drift during long-text generation.

Key facts

Read aloud

RangeSWA is a training-free method designed to improve length extrapolation in continuous diffusion language models. It addresses issues like text repetition and semantic drift during long-text generation. Continuous diffusion language models show promising potential in text generation tasks, but are prone to problems such as text repetition, semantic drift, and generation quality degradation when the generation length exceeds the training length. To address these challenges, we propose RangeSWA, a training-free length extrapolation method. The method combines local sliding-window attention, remote block summaries, and dynamic content retrieval to preserve local details and support long-range context access. We conducted experiments on multiple continuous diffusion language models, and the results show that RangeSWA can effectively reduce generative perplexity and text repetition rates in long-text generation.

Sources in this cluster