Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
arXiv cs.AI · 31 Oct 2026, 03:59 AEST
Latent Sparse Attention (LSA) is an efficient attention method designed for ultra-long context tasks. It maintains accuracy while significantly speeding up processing times for models handling over 1 million tokens.
Key facts
- Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows.
- A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention.
- We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context.
- On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks.
Read aloud
Latent Sparse Attention LSA is an efficient attention method designed for ultra-long context tasks. It maintains accuracy while significantly speeding up processing times for models handling over one million tokens. Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. We address this with Latent Sparse Attention LSA, an efficient attention method that maintains accuracy at ultra-long context. On GLM-five point three-FLASH QWEN-three point eight-27B at 10M tokens, D-LSA speeds up one attention layer by twenty two.7x seventy two.4x in prefill and four.6x forty two.5x in decode over its original sparse attention full attention, while keeping similar accuracy across a variety of tasks.
Sources in this cluster
- Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention
arXiv cs.AI · 31 Oct 2026, 03:59 AEST