The author evaluated four quantizations of the Nemotron Labs Dory 1.0 4B model—BF16, NVFP4, MXFP8 (all via vLLM) and GGUF Q5_K_M (via llama.cpp)—on an RTX 4070 Ti SUPER (Ada architecture, 16 GB VRAM, lacking native FP4 support). Generation throughput (batch = 1) was measured as 51.0 tok/s for BF16, 78.5 tok/s for per‑tensor FP8 (W8A8), 81.4 tok/s for MXFP8, and 116.8 tok/s for NVFP4, making NVFP4 2.3 × faster than BF16 while its weight footprint shrank to 3.28 GiB (vs 7.69 GiB for BF16, 4.67 GiB for FP8, 4.76 GiB for MXFP8). Prompt‑reading speed showed FP8 leading at 10,118 tok/s because Ada performs its multiplications in 8‑bit, whereas BF16 reached 7,003 tok/s, NVFP4 6,978 tok/s and MXFP8 6,061 tok/s. On the GSM8K benchmark (1 319 questions, 8 192‑token limit) scores were BF16 1249, FP8 1241, MXFP8 1250 and NVFP4 1238; all fall within ±3 points of BF16, satisfying the tie rule. Token‑by‑token fidelity on fixed English and Spanish texts indicated NVFP4 predicts BF16’s next token 89‑90 % of the time, FP8 and MXFP8 94‑96 %, with perplexity changes negligible and MXFP8 being the most faithful to BF16.
Read original
dev.to