FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
arXiv cs.AI · 17 Oct 2026, 03:22 AEST
FuTransformer is a CUDA implementation designed for Llama-style inference on NVIDIA Hopper GPUs. It significantly improves performance with faster prefill and decode processes compared to existing methods.
Key facts
- Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads.
- On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s.
- Across 8B and 70B shapes at batches 1-32, INT4 decode is 1.3-2.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point.
- Complete Llama-3.1-8B decode takes 2.8 ms per step at batch one, versus 8.1 ms for the compiled transformers implementation.
Read aloud
FuTransformer is a CUDA implementation designed for Llama-style inference on NVIDIA Hopper G P U s. It significantly improves performance with faster prefill and decode processes compared to existing methods. Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. On one H100, prefill is up to one.48x faster than compiled PyTorch FP8 and reaches one point two nine PFLOP/s. Across 8B and 70B shapes at batches one-thirty two, INT4 decode is one point three-two.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point. Complete Llama-three point one-8B decode takes two point eight ms per step at batch one, versus eight point one ms for the compiled transformers implementation.
Sources in this cluster
- FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
arXiv cs.AI · 17 Oct 2026, 03:22 AEST