Continuum Attention
arXiv cs.AI · 15 Oct 2026, 03:57 AEST
Continuum Attention (CA) is a new causal Transformer attention operator that utilizes integration over a continuous memory curve instead of discrete key and value lookups. It shows improved performance in validation negative log likelihood compared to traditional attention methods.
Key facts
- We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve.
- Each query induces an exponential relevance density along the curve.
- Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons.
- An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval.
Read aloud
Continuum Attention CA is a new causal Transformer attention operator that utilizes integration over a continuous memory curve instead of discrete key and value lookups. It shows improved performance in validation negative log likelihood compared to traditional attention methods. We introduce Continuum Attention CA, a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by zero point zero two zero zero and zero point zero four eight zero nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a zero point zero four five seven nats per byte advantage over the strongest selected softmax baseline, with a paired ninety five percent confidence interval.
Sources in this cluster
- Continuum Attention
arXiv cs.AI · 15 Oct 2026, 03:57 AEST