Benchmarking Kimi K3, a 2.8-trillion parameter Mixture-of-Experts (MoE) model, demonstrates that AMD's MI355X provides superior performance per dollar compared to NVIDIA's B300 for high-concurrency inference. Utilizing Wafer's benchmarks and rental rates from gpus.io, the MI355X achieved 48 tokens per second per dollar, significantly outperforming the B300 (33 tok/s/$) and the B200 (7 tok/s/$). While the B300 maintains a raw throughput advantage at 1,568 peak aggregate tokens per second, the MI355X's efficiency makes it highly competitive for large-scale deployments. To achieve these results on ROCm, engineers implemented two critical optimizations: adding the missing top_k_renorm_prob tensor to the ROCm sampling branch to enable speculative decoding with the DSpark draft model, and applying zero-padding to the head dimension (from 12 to 16) to trigger the AITER MLA fast-path kernel. These fixes successfully resolved prefill bottlenecks and decoding errors, boosting single-stream decode speeds by approximately 2.2x and increasing peak aggregate throughput by 18%. Although the B300 remains the preferred choice for latency-sensitive single-stream tasks, the MI355X offers a more economical solution for batch-serving workloads requiring massive memory capacity for 1-million-token context windows.

Read original