Mixture‑of‑experts (MoE) models are limited by the weight memory that must reside in RAM, as sparsity reduces compute but not the 19.5 GB footprint of a 35B‑parameter model at 4‑bit. Naïve SSD offloading fails because expert selection for layer N+1 depends on the output of layer N, preventing early reads that could overlap with computation. Edge0 introduces a per‑layer prerouter that predicts routing decisions ahead of time, enabling SSD prefetch and streaming inference that hides I/O latency. This approach closes the memory‑wall gap and allows 35B‑class MoE inference on consumer hardware.

Read original