8 GB GPU, 262K context 3.5 BPW Qwen-Flash-Next Coder 70 T/s using llama.cpp
A 6B parameter Qwen-Flash-Next model achieves 70 tokens per second with 262K context length using only 8GB VRAM and 24GB RAM, leveraging n-gram offloading to SSD and MOE (Mixture of Experts) packing with smaller experts …
→ View original source