Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training
arXiv cs.CL · 3 Nov 2026, 04:30 AEST
Sonar-MoE introduces a method for memory-efficient training of Mixture-of-Experts models. It reduces peak memory usage by 19% to 22% while maintaining training speed.
Key facts
- Most parameters of a Mixture-of-Experts (MoE) model belong to its experts, and standard training keeps every expert gradient in GPU memory until the optimizer step.
- We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away.
- When we fine-tune OLMoE-1B-7B and Granite-3.1-3B-A800M in bf16 with two expert-kernel backends, Sonar-MoE lowers peak memory by 19% to 22%.
- Each step also takes 17% to 23% less time than standard training with global-norm clipping.
- Sonar-MoE matches optimizer-in-backward over all parameters in memory and speed while still clipping the dense parameters.
Read aloud
Sonar-MoE introduces a method for memory-efficient training of Mixture-of-Experts models. It reduces peak memory usage by nineteen percent to twenty two percent while maintaining training speed. Most parameters of a Mixture-of-Experts MoE model belong to its experts, and standard training keeps every expert gradient in G P U memory until the optimizer step. We introduce Sonar-MoE, which updates each expert tensor during the backward pass and frees its gradient right away. When we fine-tune OLMoE-1B-7B and Granite-three point one-3B-A800M in bf16 with two expert-kernel backends, Sonar-MoE lowers peak memory by nineteen percent to twenty two percent. Each step also takes seventeen percent to twenty three percent less time than standard training with global-norm clipping. Sonar-MoE matches optimizer-in-backward over all parameters in memory and speed while still clipping the dense parameters.
Sources in this cluster
- Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training
arXiv cs.CL · 3 Nov 2026, 04:30 AEST