CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs
arXiv cs.AI · 16 Nov 2026, 02:47 AEST
CECAR is a new MoE inference system that improves on-device performance by optimizing expert routing and cache management. It achieves a significant speedup in processing on consumer-grade GPUs while managing memory constraints.
Key facts
- Despite growing interest in on-device LLM deployment, large-scale Mixture-of-Experts (MoE) models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident.
- We propose CECAR (Cache & Expert Co-Aware Routing), a MoE inference system that jointly considers expert routing and cache management.
- On the Qwen3-30B-A3B model, CECAR achieves 12.5 tokens/s on a consumer-grade GPU with only 16 GB of VRAM, where non-resident experts are fetched from SSD, delivering a 4.5x speedup over conventional MoE inference.
Read aloud
CECAR is a new MoE inference system that improves on-device performance by optimizing expert routing and cache management. It achieves a significant speedup in processing on consumer-grade G P U s while managing memory constraints. Despite growing interest in on-device L L M deployment, large-scale Mixture-of-Experts MoE models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident. We propose CECAR Cache & Expert Co-Aware Routing, a MoE inference system that jointly considers expert routing and cache management. On the Qwen3-30B-A3B model, CECAR achieves twelve point five tokens/s on a consumer-grade G P U with only sixteen GB of VRAM, where non-resident experts are fetched from SSD, delivering a four.5x speedup over conventional MoE inference.
Sources in this cluster
- CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs
arXiv cs.AI · 16 Nov 2026, 02:47 AEST