AI TrackerFrontier desk · AEST
Back to news
normalaitech

Subquadratic Attention with a Hierarchical Tree-Structured Indexer

arXiv cs.AI · 31 Oct 2026, 03:48 AEST

A new method called Zyphra Hierarchical Attention (ZHA) addresses the quadratic scaling issue in attention mechanisms for large language models. This approach allows for subquadratic scaling, enhancing long-context capabilities while maintaining performance on complex tasks.

Key facts

Read aloud

A new method called Zyphra Hierarchical Attention ZHA addresses the quadratic scaling issue in attention mechanisms for large language models. This approach allows for subquadratic scaling, enhancing long-context capabilities while maintaining performance on complex tasks. The quadratic scaling of attention is a fundamental bottleneck limiting the long-context capabilities of large language models L L M s. In this paper, we derive an asymptotically subquadratic sparse attention method based on a hierarchical, tree-structured indexer, which we call Zyphra Hierarchical Attention ZHA. ZHA constructs a tree of recursively compressed key summaries over the KV cache, which future queries traverse from the top down to retrieve a sparse subset of keys. We demonstrate that ZHA can be cheaply applied to existing pretrained models with only a short KL distillation training stage over a few billion tokens, and maintains performance on long-context retrieval and challenging reasoning, mathematical, and agentic coding evaluation tasks. Moreover, we show that our method is highly robust to tree geometry including very deep trees of up to six levels.

Sources in this cluster

Subquadratic Attention with a Hierarchical Tree-Structured Indexer · AI Tracker