K. L. Aadithya built tinyinfer, a 2,200-line C++ and CUDA LLM inference engine with no PyTorch or cuBLAS dependencies, achieving 235 tokens/s (77% of the 305 tokens/s memory-bandwidth ceiling) on an RTX 3050 6 GB laptop GPU running 4-bit TinyLlama-1.1B. The engine targets the memory-bound nature of autoregressive decoding, where weights are read once per token for a single multiply-add. Three optimization rounds yielded the result. Version 0.1 established a baseline using one warp per output row for GEMV kernels with 4-bit weight unpacking via an exponent trick (OR into 2²³ mantissa, subtract) and CUDA graphs to amortize ~250 launches per token, reaching 200.7 tokens/s (66% ceiling). Kernel profiling revealed bottlenecks: down-projection GEMV at 54% bandwidth and attention at 17 GB/s. Round 1 introduced autotuned multi-row GEMV (R rows per warp), selecting R ∈ {1,2,4} per matrix shape at load time; this raised TinyLlama down-projection bandwidth from 54% to 79% and Llama-2-7B Q-projection from 63% to 82%. Round 2 addressed grouped-query attention (32 Q heads, 4 KV heads). The v0.2 split-K kernel read KV once but stalled on 8 serial warp reductions per timestep, slowing short contexts (128 tokens: 10→26 µs). Version 0.3 adopted a shared-memory tile with one lane per timestep and padded rows for bank-conflict-free access, reducing reductions to two per tile and cutting 2,048-token latency from 124.6 µs to 34.2 µs (3.6× faster). Round 3 fused QKV and gate/up projections into two launches, lifting throughput to 235.2 tokens/s. A regression on non-grouped models (Llama-2-7B: 269 vs 219 µs) was fixed by gating the tiled kernel on grouped heads. Correctness was verified against a CPU reference and Hugging Face across 16/8/4-bit formats (max 2.5e-4 relative error, 100% argmax match). Next steps target prompt processing via batched GEMM to leverage tensor cores.

Read original