LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models
arXiv cs.CL · 5 Nov 2026, 03:52 AEST
LooM introduces Loop-invariant Memory to enhance causal encoder-decoder language models by addressing key limitations of Looped Transformers. This approach allows for independent looping of decoder layers, improving performance without increasing memory usage.
Key facts
- Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training.
- In a causal encoder-decoder (CED), LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value (KV) entry for a token only on its first visit, so the cache does not grow with the loop count.
- With enough loops, LooM outperforms a single-pass baseline trained on the same data on language modeling, zero-shot accuracy and long-context perplexity, while using the same cache size.
Read aloud
LooM introduces Loop-invariant Memory to enhance causal encoder-decoder language models by addressing key limitations of Looped Transformers. This approach allows for independent looping of decoder layers, improving performance without increasing memory usage. Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training. In a causal encoder-decoder CED, LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value KV entry for a token only on its first visit, so the cache does not grow with the loop count. With enough loops, LooM outperforms a single-pass baseline trained on the same data on language modeling, zero-shot accuracy and long-context perplexity, while using the same cache size.
Sources in this cluster
- LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models
arXiv cs.CL · 5 Nov 2026, 03:52 AEST