Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
arXiv cs.AI · 24 Oct 2026, 02:46 AEST
Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs. This work introduces Layer-wise Error Modeling Search (LEMS) and KFAC-SVD to address key limitations in existing methods.
Key facts
- Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of LLMs.
- In this work, we address these limitations by introducing Layer-wise Error Modeling Search (LEMS) and KFAC-SVD.
- LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias.
- Across Mistral, Qwen3, and Llama 3 model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to 4.8 p.p.
- KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches.
Read aloud
Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of L L M s. This work introduces Layer-wise Error Modeling Search LEMS and KFAC-SVD to address key limitations in existing methods. Low-rank SVD-based compression offers a powerful strategy to reduce the computational costs of L L M s. In this work, we address these limitations by introducing Layer-wise Error Modeling Search LEMS and KFAC-SVD. LEMS advances rank allocation by introducing a layer-wise error surrogate that integrates local and global layer importance alongside a propagation bias. Across Mistral, Qwen3, and Llama three model families, we show that LEMS consistently outperforms existing search strategies, delivering significant zero-shot accuracy gains of up to four point eight p.p. KFAC-SVD improves decomposition quality by utilizing token-wise statistics, mitigating the rank deficiency observed in prior Fisher-based SVD approaches.
Sources in this cluster
- Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
arXiv cs.AI · 24 Oct 2026, 02:46 AEST