Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models
arXiv cs.AI · 11 Nov 2026, 03:53 AEST
Local language models can optimize memory usage by adjusting weight precision and option-order averaging. The study evaluates the trade-offs between these methods using simulated weight quantization on small models.
Key facts
- Local language models can save memory by storing weights in fewer bits, and they can reduce sensitivity to the order of answer options by averaging a few reorderings.
- On Qwen3-0.6B and Qwen3-1.7B with simulated round-to-nearest weight quantization (8, 6, 4 and 3 bits, fp16 reference), 6 bits lose no accuracy against fp16.
- At 4 bits the answer-slot skew rises from 0.15 to 0.53 (0.6B) and from 0.07 to 0.12 (1.7B); Gaussian noise at the same weight SNR gives 0.15 and 0.37 (two draws) and 0.19 (one draw).
- Averaging four cyclic orders recovers +12.6 points at 4 bits for 0.6B (+5.3 at fp16).
- In cost-matched pairs, extra orders beat extra bits only when the weights have 6 or more bits.
Read aloud
Local language models can optimize memory usage by adjusting weight precision and option-order averaging. The study evaluates the trade-offs between these methods using simulated weight quantization on small models. Local language models can save memory by storing weights in fewer bits, and they can reduce sensitivity to the order of answer options by averaging a few reorderings. On Qwen3-zero.6B and Qwen3-one.7B with simulated round-to-nearest weight quantization eight, six, four and three bits, fp16 reference, six bits lose no accuracy against fp16. At four bits the answer-slot skew rises from zero point one five to zero point five three zero.6B and from zero point zero seven to zero point one two one.7B; Gaussian noise at the same weight SNR gives zero point one five and zero point three seven two draws and zero point one nine one draw. Averaging four cyclic orders recovers +twelve point six points at four bits for zero.6B +five point three at fp16. In cost-matched pairs, extra orders beat extra bits only when the weights have six or more bits.
Sources in this cluster
- Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models
arXiv cs.AI · 11 Nov 2026, 03:53 AEST