A 6B parameter Qwen-Flash-Next model achieves 70 tokens per second with 262K context length using only 8GB VRAM and 24GB RAM, leveraging n-gram offloading to SSD and MOE (Mixture of Experts) packing with smaller experts in VRAM. The approach exploits the model's architecture by keeping most MOE weights in system RAM while placing the smallest experts in VRAM for performance, requiring users to use fast SSDs for optimal n-gram access. Users with approximately 32GB total system memory can also run the model with reduced context sizes by adjusting launch parameters.

Read original