tandem.cpp is a patch set for llama.cpp that accelerates Mixture-of-Experts (MoE) models exceeding VRAM capacity by keeping frequently used experts on the GPU while offloading others to the CPU. On a consumer system (i5-12400F, 64 GB RAM, RTX 4090), it achieves 97.3 tokens/second with Qwen3.8-Flash-Next (IQ3_S + MTP), a 3.3× speedup over standard llama.cpp. The optimization targets single-user decoding and works with any MoE model supported by llama.cpp.

Read original