Avoid using 4‑bit (Q4) quantization for the key‑value cache in LLMs, as it causes early and compounding degradation of model quality. 8‑bit (Q8) quantization is acceptable when memory is limited and the context length stays below ~128k tokens, but Q4 keys should not be used.