This technical report details the deployment of Google's Gemma 4 E2B model on AWS EC2 G5g instances, which utilize an ARM64 Graviton2 host paired with an NVIDIA T4G GPU (Turing architecture, compute capability 7.5). The implementation required overcoming several hardware-software mismatches, most notably the lack of published aarch64 builds for SM 7.5 and the absence of the CUDA toolkit in standard AWS Deep Learning AMIs. To resolve this, the author built vLLM v0.27.2rc0 from source using the AWS ARM64 GPU DLAMI (PyTorch 2.12, CUDA 13.2) and manually installed the NVIDIA sbsa cuda-toolkit. A critical technical hurdle involved Gemma 4's heterogeneous attention heads (256-wide sliding and 512-wide full attention), which exceeded the T4G's 64 KiB shared memory limit when using Triton attention kernels. This was resolved by patching the vLLM Triton kernel to shrink the KV tile size and reduce the software pipeline to a single stage. Following these optimizations, the system achieved a single-stream greedy throughput of 43.1 tokens per second and a KV cache capacity of 329,579 tokens. Performance was limited by the T4G's GDDR6 memory bandwidth, which measured at 277 GB/s for streaming reads.

Read original