LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers
arXiv cs.AI · 4 Nov 2026, 04:16 AEST
LoopVQ is a KV cache vector quantization method designed for looped transformers, which are efficient for language modeling. It significantly reduces memory usage while maintaining performance in various tasks.
Key facts
- Looped transformers are a parameter-efficient alternative to standard transformers for language modeling.
- While this architecture effectively reduces parameter memory, each loop step still stores its own keys and values, so the key-value (KV) cache grows linearly with the number of loop steps and becomes the main memory bottleneck.
- At 0.5 bits per element, LoopVQ outperforms existing KV cache vector quantization methods for standard LLMs on general, long chain-of-thought reasoning, and long-context retrieval tasks with Ouro and Huginn models.
- We also implement LoopVQ in SGLang, where it nearly halves the decode latency at a 32K-token context and raises the peak serving throughput by up to 4.3× and 6.1× on the 1.4B and 2.6B Ouro models compared to the BF16 baseline.
Read aloud
LoopVQ is a KV cache vector quantization method designed for looped transformers, which are efficient for language modeling. It significantly reduces memory usage while maintaining performance in various tasks. Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. While this architecture effectively reduces parameter memory, each loop step still stores its own keys and values, so the key-value KV cache grows linearly with the number of loop steps and becomes the main memory bottleneck. At zero point five bits per element, LoopVQ outperforms existing KV cache vector quantization methods for standard L L M s on general, long chain-of-thought reasoning, and long-context retrieval tasks with Ouro and Huginn models. We also implement LoopVQ in SGLang, where it nearly halves the decode latency at a 32K-token context and raises the peak serving throughput by up to four point three× and six point one× on the one.4B and two.6B Ouro models compared to the BF16 baseline.
Sources in this cluster
- LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers
arXiv cs.AI · 4 Nov 2026, 04:16 AEST