AI TrackerFrontier desk · AEST
Back to news
normalaitech

Bits or permutations? Spending a fixed inference budget on weight precision or on option-order averaging in small local language models

arXiv cs.AI · 11 Nov 2026, 03:53 AEST

Local language models can optimize memory usage by adjusting weight precision and option-order averaging. The study evaluates the trade-offs between these methods using simulated weight quantization on small models.

Key facts

Read aloud

Local language models can optimize memory usage by adjusting weight precision and option-order averaging. The study evaluates the trade-offs between these methods using simulated weight quantization on small models. Local language models can save memory by storing weights in fewer bits, and they can reduce sensitivity to the order of answer options by averaging a few reorderings. On Qwen3-zero.6B and Qwen3-one.7B with simulated round-to-nearest weight quantization eight, six, four and three bits, fp16 reference, six bits lose no accuracy against fp16. At four bits the answer-slot skew rises from zero point one five to zero point five three zero.6B and from zero point zero seven to zero point one two one.7B; Gaussian noise at the same weight SNR gives zero point one five and zero point three seven two draws and zero point one nine one draw. Averaging four cyclic orders recovers +twelve point six points at four bits for zero.6B +five point three at fp16. In cost-matched pairs, extra orders beat extra bits only when the weights have six or more bits.

Sources in this cluster