Drawing from Philip Kiely's "Inference Engineering: Serving Generative AI in Production," this article structures the discipline around three layers: Runtime (single-instance CUDA kernels, PyTorch, engines like vLLM, SGLang, TensorRT-LLM), Infrastructure (multi-cloud routing, autoscaling, container orchestration), and Tooling (observability, SDKs, benchmarking). It maps the automation boundary: engines handle in-flight continuous batching, PagedAttention KV cache paging, graph compilation, and NCCL collectives, while engineers must manually tune multi-replica load balancing, KV cache memory fractions, CPU offloading, custom CUDA kernels (FlashAttention, DeepGEMM), post-training calibration via ModelOpt, EAGLE speculator head training, parallel topology selection (TP/PP/EP/CP), and pinned dependency trees. Core metrics are compute-bound Time to First Token (TTFT) during Prefill and memory-bandwidth-bound Tokens Per Second (TPS) during Decode. Optimization techniques include continuous batching at token granularity, prefix caching to cut TTFT from milliseconds to tens, quantization (FP8/INT4/MXFP4) for prefill FLOPS and decode bandwidth gains, and speculative decoding via draft-target, EAGLE (trained head on hidden states), or n-gram methods. Parallelism strategies distribute models exceeding VRAM: TP splits matmuls (needs NVLink), PP passes activations sequentially, EP routes MoE experts, CP divides long contexts via Ring Attention. Disaggregated serving separates prefill and decode workers onto independent clusters connected by InfiniBand/Ethernet. Engine comparison shows vLLM for broad hardware support and ease (pip, AMD/Intel/TPU), SGLang for MoE throughput and frontend flexibility, TensorRT-LLM for peak NVIDIA performance (Docker, config.yaml). Production practices mandate pinned container dependencies (exact versions), local weight/engine binary caching to eliminate cold-start latency, and cost-effective 4-step canary deployments (2–5% traffic) over blue/green. A Baseten case study notes 77 benchmarked configurations doubled throughput for code completion.

Read original