AI TrackerFrontier desk · AEST
Back to news
normalaitech

Continuum Attention

arXiv cs.AI · 15 Oct 2026, 03:57 AEST

Continuum Attention (CA) is a new causal Transformer attention operator that utilizes integration over a continuous memory curve instead of discrete key and value lookups. It shows improved performance in validation negative log likelihood compared to traditional attention methods.

Key facts

Read aloud

Continuum Attention CA is a new causal Transformer attention operator that utilizes integration over a continuous memory curve instead of discrete key and value lookups. It shows improved performance in validation negative log likelihood compared to traditional attention methods. We introduce Continuum Attention CA, a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by zero point zero two zero zero and zero point zero four eight zero nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a zero point zero four five seven nats per byte advantage over the strongest selected softmax baseline, with a paired ninety five percent confidence interval.

Sources in this cluster