AI TrackerFrontier desk · AEST
Back to news
normalaitech

Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

arXiv cs.AI · 31 Oct 2026, 03:59 AEST

Latent Sparse Attention (LSA) is an efficient attention method designed for ultra-long context tasks. It maintains accuracy while significantly speeding up processing times for models handling over 1 million tokens.

Key facts

Read aloud

Latent Sparse Attention LSA is an efficient attention method designed for ultra-long context tasks. It maintains accuracy while significantly speeding up processing times for models handling over one million tokens. Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. We address this with Latent Sparse Attention LSA, an efficient attention method that maintains accuracy at ultra-long context. On GLM-five point three-FLASH QWEN-three point eight-27B at 10M tokens, D-LSA speeds up one attention layer by twenty two.7x seventy two.4x in prefill and four.6x forty two.5x in decode over its original sparse attention full attention, while keeping similar accuracy across a variety of tasks.

Sources in this cluster