This repository provides a production-ready software stack to run the 304B-parameter DeepSeek-V4-Flash-0731 model on a single AMD MI300X GPU. To achieve this, the author implemented several critical fixes for the CDNA3 architecture, including addressing FP8 scale errors by implementing the AMD FNUZ variant of E4M3, fixing MXFP4 padded-lane routing bugs, and adding expert-activation clamps to ensure bit-exact correctness. The deployment utilizes a hybrid KV cache strategy consisting of a 16 GB fp8_ds_mla GPU pool and a 96 GiB native CPU tier, supporting contexts of up to 384K tokens. Performance optimizations include custom JIT-compiled gfx942 kernels for asymmetric INT8 MoE, OPUS sparse-prefill, and exact BF16 SwiGLU. Benchmarks show a 2.19× improvement in uncached C1 prefill, reaching 11.69K tok/s. For decoding, the implementation uses DSpark-7 speculative decoding with static K=7, achieving 152.6 tok/s aggregate (158.8 tok/s median) in single-stream scenarios and 1,278 tok/s aggregate during 64-stream bursts. The stack relies on a pinned vLLM ROCm nightly (0.26.1rc1) and AITER 0.1.19, utilizing a contention-aware scheduler to manage prefill budgets and prevent latency regressions during multi-user workloads.

Read original