A Reddit user quantified the impact of KV cache quantization on Qwen3.8 27B using Kullback-Leibler divergence (KLD) to measure output distribution shifts. Testing 35 weight quantizations across llama.cpp, exllamav3, and vLLM with 16-bit, 8-bit, and lower precision KV caches, the analysis aims to resolve community debates ranging from "never quantize" to "4-bit is fine." Results provide empirical data on how quantization levels affect model behavior across inference engines.

Read original