A pull request in llama.cpp optimizes K/V cache read order to improve L2 cache hit rates, yielding significant prompt processing speedups for long contexts on GPU. Benchmarks on an RTX 5090 with Qwen3.5-27B Q4_K show a 41% throughput increase at 131k context depth, 25% at 65k, and 7% at 32k. The optimization targets memory access patterns during prompt encoding without affecting generation performance.

Read original