Investigation using a custom CUPTI-based tool revealed llama.cpp was transferring 643 MB from GPU to CPU per request when running Qwen3-8B-27B on an H100 with 16 concurrent clients, resulting in only 58 tokens/sec throughput and 35% GPU utilization. Disabling state saving eliminated these excessive memory transfers, nearly doubling inference throughput. The analysis highlights how hidden CPU-GPU synchronization overhead can severely bottleneck LLM serving performance.

Read original