AI TrackerFrontier desk · AEST
Back to news
normalaitech

Multi-State Linear Attention with Generalized Second-Order Routing

arXiv cs.CL · 28 Oct 2026, 03:55 AEST

A new method called MERLIN has been proposed to enhance Multi-State Linear Attention in long-sequence modeling. This approach improves the efficiency of Large Language Models by incorporating variance in alignment for better content-aware aggregation.

Key facts

Read aloud

A new method called MERLIN has been proposed to enhance Multi-State Linear Attention in long-sequence modeling. This approach improves the efficiency of Large Language Models by incorporating variance in alignment for better content-aware aggregation. Self-attention plays a key role in long-sequence modeling, but its quadratic time complexity fundamentally limits the scaling of Large Language Models. Recently, Multi-State Linear Attention MSLA mechanisms have been proposed to make sequence modeling more efficient by splitting the context into segments and compressing each in its own memory state. Experiments on multi-state variants of Gated DeltaNet-one.3B and Mamba-two-780M show that MERLIN consistently outperforms existing aggregation methods on various long-context tasks. In particular, MERLIN outperforms baselines on RULER needle-in-a-haystack and long-context QA +eight point three percent for Gated DeltaNet-one.3B and +seven point zero percent for Mamba-two-780M and diverse subtasks in LongBench +thirteen point five percent and +twelve point zero percent.

Sources in this cluster