Mixture-of-experts models frequently exceed a single accelerator’s memory, and offloading experts to host memory creates a severe PCIe I/O bottleneck because data‑dependent routing keeps transfers on the critical path. Specter mitigates this by using quantization to combine speculative decoding with expert offloading, enabling a lightweight quantized draft model to run efficiently on‑device while preserving high‑fidelity expert activation patterns. This approach reduces critical‑path transfers and accelerates inference for large MoE models.
Read original
reddit/r/LocalLLaMA