Low-Bit N-Gram Embeddings for Language Models
arXiv cs.AI · 23 Oct 2026, 03:56 AEST
Recent research explores low-bit n-gram embeddings to reduce memory costs in language models. The study finds that four-bit tables can save approximately 75% of memory while maintaining over 99% of performance improvements.
Key facts
- Many recent open-weight language models use n-gram embedding tables to improve performance.
- We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables.
- Across nanochat-, Llama-, and GPT-2-family models trained from scratch, four-bit tables save approximately 75% of table memory while preserving more than 99% of the validation-loss improvement from adding the tables.
- These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.
Read aloud
Recent research explores low-bit n-gram embeddings to reduce memory costs in language models. The study finds that four-bit tables can save approximately seventy five percent of memory while maintaining over ninety nine percent of performance improvements. Many recent open-weight language models use n-gram embedding tables to improve performance. We propose storing n-gram embeddings at low precision and systematically investigate different quantization methods and bit allocations across tables. Across nanochat-, Llama-, and GPT-two-family models trained from scratch, four-bit tables save approximately seventy five percent of table memory while preserving more than ninety nine percent of the validation-loss improvement from adding the tables. These findings establish low-bit precision as a practical way to reduce the memory requirements of n-gram embeddings and use the saved capacity to improve model performance.
Sources in this cluster
- Low-Bit N-Gram Embeddings for Language Models
arXiv cs.AI · 23 Oct 2026, 03:56 AEST