A user with a multi-GPU setup (6× RTX PRO 6000, 8× RTX 5090, 4× RTX 3090) on a Threadripper PRO 5995WX platform seeks advice on scaling system memory for MoE model offloading. They compare expanding to 512GB DDR4-2933 (~188 GB/s bandwidth, ~€4k) versus migrating to a Threadripper PRO 9985WX/WRX90 platform with 512GB DDR5-6400 (~410 GB/s bandwidth). The goal is running ~1 trillion parameter MoE models at FP8/Q8 quantization with routed experts offloaded to system RAM.

Read original