A user achieved 96.32 tokens/s for coding tasks using Qwen 3.8 with 8-bit FP quantization and Multi-Token Prediction (MTP) on $1500 hardware featuring 24 GPUs. Token-residual distribution across GPUs takes 128.0 µs per layer, indicating near-peak hardware utilization. The primary bottleneck is now an older RTX 3070 handling attention, while natural text generation runs at 69 tokens/s.
Read original
reddit/r/LocalLLM