Janas-LLM is a dependency-free C inference engine that runs MoE models exceeding system RAM by streaming experts from NVMe storage with direct I/O and caching frequently used experts. It achieves 26 tokens/second on Qwen3-Next-80B (48 GB) using only 32 GB RAM, significantly outperforming llama.cpp's 3.0 tok/s. The project includes an OpenAI-compatible server, MCP tools, and vision/speech capabilities under GPL-3.0.
Read original
reddit/r/LocalLLM