A developer reports achieving 7K tokens/s prefill and 814 tokens/s decode throughput running Qwen3.8 27B FP8 on four Radeon AI PRO R9700 GPUs, each capped at 210W for standard outlet compatibility. The setup uses BF16 KV cache and Multi-Token Prediction (MTP) with a custom inference engine supporting on-device training and routing. Latency metrics show a mean TTFT of 4.6s and TPOT of 9.93ms, with end-to-end latency averaging 9.68s. This demonstrates high-efficiency local LLM inference on consumer-accessible AMD hardware.

Read original