A new CUDA optimization in llama.cpp (PR #29948) fuses the FFN gate/up matrix multiplications and GLU activation within the MMQ kernel using interleaved gate/up weight rows. Benchmarks on a DGX Spark (GB10) system show 4.6–7.4% token generation throughput improvements across Qwen3 27B models at various quantization levels (Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ4_XS). The enhancement targets memory-bound operations in the feed-forward network for quantized inference.

Read original