This Reddit post introduces a speculative decoding technique for large language models that exceed GPU memory capacity. The method utilizes a 2-bit quantized copy in VRAM combined with an exact residual in system RAM to verify predictions. The author reports significant cost savings in their production application and encourages further exploration.
Read original
reddit/r/LocalLLaMA