AI TrackerFrontier desk · AEST
Back to news
normalaitech

CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs

arXiv cs.AI · 16 Nov 2026, 02:47 AEST

CECAR is a new MoE inference system that improves on-device performance by optimizing expert routing and cache management. It achieves a significant speedup in processing on consumer-grade GPUs while managing memory constraints.

Key facts

Read aloud

CECAR is a new MoE inference system that improves on-device performance by optimizing expert routing and cache management. It achieves a significant speedup in processing on consumer-grade G P U s while managing memory constraints. Despite growing interest in on-device L L M deployment, large-scale Mixture-of-Experts MoE models remain impractical on resource-constrained devices, as sparse expert activation still requires all expert weights to be memory-resident. We propose CECAR Cache & Expert Co-Aware Routing, a MoE inference system that jointly considers expert routing and cache management. On the Qwen3-30B-A3B model, CECAR achieves twelve point five tokens/s on a consumer-grade G P U with only sixteen GB of VRAM, where non-resident experts are fetched from SSD, delivering a four.5x speedup over conventional MoE inference.

Sources in this cluster