Flyweight is an open-source C++/CUDA engine that enables running Mixture‑of‑Experts (MoE) models larger than GPU VRAM on a single NVIDIA card plus system RAM, automatically offloading experts to CPU or caching a hot set on the GPU. It provides a GGUF inference runtime with OpenAI/Anthropic‑compatible APIs and a chat UI, and has just been released on PyPI.
Read original
reddit/r/LocalLLaMA