AI TrackerFrontier desk · AEST
Back to news
normalaitech

Multi-View MoE: Letting Every Router Shape Expert Computation

arXiv cs.CL · 30 Oct 2026, 03:45 AEST

Multi-View MoE enhances expert selection in sparse mixture-of-experts models by utilizing multiple routers. This approach allows for improved computation efficiency while maintaining a fixed expert budget.

Key facts

Read aloud

Multi-View MoE enhances expert selection in sparse mixture-of-experts models by utilizing multiple routers. This approach allows for improved computation efficiency while maintaining a fixed expert budget. Sparse mixture-of-experts MoE models expand parameter capacity by activating a small subset of experts per token. Dense evaluation of all experts reveals weak agreement between single-router rankings and expert-utility evaluation rankings, especially in middle and deep layers. We propose Multi-View MoE to improve expert selection with multiple routers and reuse their preferences to modulate expert computation under a fixed expert budget. On a twenty four-layer 10B-A1B model trained on 708B tokens, full Multi-View MoE improves the nine-task downstream mean by one point one eight percentage points over the baseline.

Sources in this cluster