Multi-View MoE: Letting Every Router Shape Expert Computation
arXiv cs.CL · 30 Oct 2026, 03:45 AEST
Multi-View MoE enhances expert selection in sparse mixture-of-experts models by utilizing multiple routers. This approach allows for improved computation efficiency while maintaining a fixed expert budget.
Key facts
- Sparse mixture-of-experts (MoE) models expand parameter capacity by activating a small subset of experts per token.
- Dense evaluation of all experts reveals weak agreement between single-router rankings and expert-utility evaluation rankings, especially in middle and deep layers.
- We propose Multi-View MoE to improve expert selection with multiple routers and reuse their preferences to modulate expert computation under a fixed expert budget.
- On a 24-layer 10B-A1B model trained on 708B tokens, full Multi-View MoE improves the nine-task downstream mean by 1.18 percentage points over the baseline.
Read aloud
Multi-View MoE enhances expert selection in sparse mixture-of-experts models by utilizing multiple routers. This approach allows for improved computation efficiency while maintaining a fixed expert budget. Sparse mixture-of-experts MoE models expand parameter capacity by activating a small subset of experts per token. Dense evaluation of all experts reveals weak agreement between single-router rankings and expert-utility evaluation rankings, especially in middle and deep layers. We propose Multi-View MoE to improve expert selection with multiple routers and reuse their preferences to modulate expert computation under a fixed expert budget. On a twenty four-layer 10B-A1B model trained on 708B tokens, full Multi-View MoE improves the nine-task downstream mean by one point one eight percentage points over the baseline.
Sources in this cluster
- Multi-View MoE: Letting Every Router Shape Expert Computation
arXiv cs.CL · 30 Oct 2026, 03:45 AEST