The study investigates remote KV‑cache tiers in a disaggregated GLM‑5.3 (FP8 MoE with multi‑head latent attention) deployment on 8×B300 nodes using prefill context parallelism (PCP8). Enabling the remote tier (LMCache with a Valkey store) left median time‑to‑first‑token unchanged but degraded the 16 k‑token p99 from 3.96 s to 17.3 s. Isolated replay of a production stream showed that a remote hit incurred 0.88 s latency versus 0.31 s for recomputation, and synchronous loading turned a hit‑heavy workload into a queue with p99 = 83.32 s while GPUs were only 7 % busy. A two‑term cost model reveals that a hit is worthwhile only when the tier supplies bandwidth above B* = sR = 2.53 GB/s and that a fixed per‑hit engine overhead T0 ≈ 0.33 s makes hits on prompts shorter than ~17 k tokens slower than recompute regardless of bandwidth. Guided by the model, the authors (i) sharded the MLA KV store across PCP ranks, proving that the minimum‑over‑ranks lookup returns the longest cached prefix and that the load protocol never desynchronizes collectives; (ii) partitioned the remote tier by rank—running one Valkey instance per rank without coordination—boosting GET bandwidth 6.3× to 10.76 GB/s; (iii) introduced asynchronous loading for the sharded store; and (iv) corrected three client bugs that caused unbounded stalls. Consequently, the hit‑heavy stream achieved a p99 latency of 0.67 s (down from 83.32 s) with 98.5 % of tokens served remotely, while the sharded store increased prefill throughput by 14 % and, together with an upstream kernel fix, by 34 %. All changes have been merged upstream to LMCache.

Read original