AI TrackerFrontier desk · AEST
Back to news
normalaitech

MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control

arXiv cs.AI · 28 Oct 2026, 03:56 AEST

MuonGS is a new approach for pretraining language models that utilizes relaxed weight-norm control. It demonstrates improved performance over previous methods, achieving lower validation perplexity and faster training speeds.

Key facts

Read aloud

MuonGS is a new approach for pretraining language models that utilizes relaxed weight-norm control. It demonstrates improved performance over previous methods, achieving lower validation perplexity and faster training speeds. Weight decay and explicit weight-norm constraints improve optimizer performance in language model pretraining, yet a unified understanding of why such control helps and how much is needed remains incomplete. We show that vanilla Muon can accumulate only logarithmically, potentially limiting exploration and helping explain its slower empirical progress, whereas MuonW and MuonH guarantee at least linear accumulation under the stated conditions. Based on this insight, we propose MuonGS, which uses Gram--Schmidt GS projection to remove the radial component of each Muon update without fixing the weight norm. In pretraining experiments with 160M- and 400M-parameter dense models as well as a 135M active- and 1B total- parameter MoE model, MuonGS achieves lower validation perplexity than MuonH at long training horizons.

Sources in this cluster