BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
arXiv cs.AI · 7 Nov 2026, 04:13 AEST
BitDecoding-FP4 introduces an algorithm-system co-design for efficient native FP4 KV decoding. It significantly improves accuracy and throughput in long-context LLM serving.
Key facts
- The key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving.
- We present BitDecoding-FP4, an algorithm--system co-design for accurate and efficient native FP4 KV decoding.
- Across dense and MoE models with GQA and MLA, BitDecoding-FP4 stays within 0.5 points of BF16 KV accuracy with NVFP4 (1 point with MXFP4), and tracks BF16 KV in RL training rollouts.
- In end-to-end serving on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, it delivers 2.4-3.0× higher peak serving throughput than BF16 KV.
Read aloud
BitDecoding-FP4 introduces an algorithm-system co-design for efficient native FP4 KV decoding. It significantly improves accuracy and throughput in long-context L L M serving. The key-value KV cache dominates memory capacity and bandwidth in long-context L L M serving. We present BitDecoding-FP4, an algorithm--system co-design for accurate and efficient native FP4 KV decoding. Across dense and MoE models with GQA and MLA, BitDecoding-FP4 stays within zero point five points of BF16 KV accuracy with NVFP4 one point with MXFP4, and tracks BF16 KV in RL training rollouts. In end-to-end serving on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, it delivers two point four-three point zero× higher peak serving throughput than BF16 KV.
Sources in this cluster
- BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
arXiv cs.AI · 7 Nov 2026, 04:13 AEST