AI TrackerFrontier desk · AEST
Back to news
normalaitech

LoopVQ: Sub 1-bit KV Cache Quantization for Looped Transformers

arXiv cs.AI · 4 Nov 2026, 04:16 AEST

LoopVQ is a KV cache vector quantization method designed for looped transformers, which are efficient for language modeling. It significantly reduces memory usage while maintaining performance in various tasks.

Key facts

Read aloud

LoopVQ is a KV cache vector quantization method designed for looped transformers, which are efficient for language modeling. It significantly reduces memory usage while maintaining performance in various tasks. Looped transformers are a parameter-efficient alternative to standard transformers for language modeling. While this architecture effectively reduces parameter memory, each loop step still stores its own keys and values, so the key-value KV cache grows linearly with the number of loop steps and becomes the main memory bottleneck. At zero point five bits per element, LoopVQ outperforms existing KV cache vector quantization methods for standard L L M s on general, long chain-of-thought reasoning, and long-context retrieval tasks with Ouro and Huginn models. We also implement LoopVQ in SGLang, where it nearly halves the decode latency at a 32K-token context and raises the peak serving throughput by up to four point three× and six point one× on the one.4B and two.6B Ouro models compared to the BF16 baseline.

Sources in this cluster