NVIDIA demonstrates serving the 753-billion-parameter GLM-5.3 NVFP4 mixture-of-experts model across two GB300 DGX Stations connected via ConnectX-8 RDMA, using vLLM with tensor parallelism (TP=2) to split the ~464 GB checkpoint across both GPUs. Each station loads approximately 218 GiB of weights into its 252 GB HBM3e memory, with CPU offload disabled. The vLLM v0.30.0 container runs with NCCL 2.30.7 over RoCE, configured with --gpu-memory-utilization 0.93, --max-model-len 98304, FP8 KV-cache dtype, prefix caching, and multi-token prediction (MTP) speculative decoding drafting four tokens per step. The kernel backend uses FlashInfer-CUTLASS for MoE operations, while CUDA graphs capture decode sizes up to 80. On SPEED-Bench Coding prompts (80 conversations, 89 turns), overall output throughput peaks at 607.8 tok/s at concurrency 16, with mean request latency of 136.5 s and TTFT of 482.9 ms. At concurrency 32, throughput drops to 385.0 tok/s and latency rises to 453.5 s. Per-user throughput falls from 115.3 tok/s at C1 to 13.0 tok/s at C32. The service exposes a single OpenAI-compatible API on Station A, enabling local inference on sensitive data without per-token costs.

Read original