Rethinking Routing Exploration in Modern Mixture-of-Experts
arXiv cs.AI · 16 Oct 2026, 03:58 AEST
Mixture-of-experts (MoE) models are increasingly used to scale model parameters, combining fine-grained expert segmentation with shared experts. The proposed MirrorMoE addresses the exploration dilemma by optimizing expert routing for improved performance.
Key facts
- Mixture-of-experts (MoE) models have become an important approach for scaling model parameters.
- Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts.
- We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations.
- Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice.
- MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts.
Read aloud
Mixture-of-experts MoE models are increasingly used to scale model parameters, combining fine-grained expert segmentation with shared experts. The proposed MirrorMoE addresses the exploration dilemma by optimizing expert routing for improved performance. Mixture-of-experts MoE models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts.
Sources in this cluster
- Rethinking Routing Exploration in Modern Mixture-of-Experts
arXiv cs.AI · 16 Oct 2026, 03:58 AEST