Researchers propose a dynamic precision scheme for local LLM inference that stores 8-bit weights but decodes primarily with their top 4 bits, escalating to full 8-bit precision only when the low-precision output distribution indicates uncertainty. Evaluated on Qwen3-0.6B using GSM8K, the method quantifies cost in weight bytes read per token. In a replay analysis of 36,724 steps, low-precision entropy predicts argmax flips relative to high precision with an AUC of 0.93. An entropy-based gate operating at a 15% escalation budget captures 78% of flips, while a top-two margin gate captures 94%, both significantly outperforming the 15% capture rate of a random gate. In live evaluation across 150 problems, all-high precision scores 64.0% versus 55.3% for all-low (McNemar p=0.053). An entropy gate with a 16% escalation rate achieves 62.7% accuracy at 87% of the all-high weight-byte cost. Its paired difference versus all-high is -1.3 points (95% CI: -8.0 to +5.3, p=0.080) and versus all-low is +7.3 points (CI: +0.0 to +14.7, p=0.080), indicating the live data does not statistically separate the gate from either baseline. Verified speculation matches this byte efficiency with zero KL divergence. The study notes that a live random control, thinking-mode phase rule, second model, and live KL measurements were not executed.
Read original
huggingface/daily-papers