A modification to llama.cpp has been developed that enables the KV cache to exceed physical VRAM capacity by adaptively streaming portions between system RAM and GPU memory. This technique allows running Qwen 3.8 27B with a 262K context window on a 16GB VRAM GPU (RTX 5070 Ti). The implementation is available as a public GitHub branch for community testing across different hardware configurations.

→ View original source