AI TrackerFrontier desk · AEST
Back to news
normalaitech

Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training

arXiv cs.CL · 3 Nov 2026, 04:30 AEST

Sonar-MoE introduces a method for memory-efficient training of Mixture-of-Experts models. It reduces peak memory usage by 19% to 22% while maintaining training speed.

Key facts

Read aloud

Sonar-MoE introduces a method for memory-efficient training of Mixture-of-Experts models. It reduces peak memory usage by nineteen percent to twenty two percent while maintaining training speed. Most parameters of a Mixture-of-Experts MoE model belong to its experts, and standard training keeps every expert gradient in G P U memory until the optimizer step. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. When we fine-tune OLMoE-1B-7B and Granite-three point one-3B-A800M in bf16 with two expert-kernel backends, Sonar-MoE lowers peak memory by nineteen percent to twenty two percent. Each step also takes seventeen percent to twenty three percent less time than standard training with global-norm clipping. Sonar-MoE matches optimizer-in-backward over all parameters in memory and speed while still clipping the dense parameters.

Sources in this cluster