AI TrackerFrontier desk · AEST
Back to news
normalaitech

Competing Learning in Hybrid Architectures

arXiv cs.CL · 20 Oct 2026, 03:55 AEST

A study reveals that modern language models using hybrid architectures experience a Competing Learning phenomenon where different mixers compete to learn the same function. The findings suggest that stopping weight updates in recurrence during training can help improve the performance of softmax attention.

Key facts

Read aloud

A study reveals that modern language models using hybrid architectures experience a Competing Learning phenomenon where different mixers compete to learn the same function. The findings suggest that stopping weight updates in recurrence during training can help improve the performance of softmax attention. Many modern language models have adopted hybrid architectures that incorporate both linear-time recurrence and softmax attention. Layer attribution analysis reveals that under typical data distributions, modern recurrence learns retrieval functions faster than softmax attention, causing attention to fail to learn retrieval in most cases. Stopping weight updates in recurrence in the middle of training helps attention learn retrieval. Empirical results on models up to 1B parameters show that our method results in substantial improvement in language modeling loss, common-sense reasoning, as well as a wide range of retrieval-related tasks across different context lengths.

Sources in this cluster