A pull request introduces moe-cache, an LRFU/MQ-like hybrid cache using ClassLRU to prioritize frequently used experts via decaying usage history. Benchmarks show a 6% decode improvement on QFN and 7% on GLM models over SLRU, with 8% fewer uploads. The implementation enables an auto mode for llama_moe_cache and awaits the merge of PR #30966.
Read original
reddit/r/LocalLLaMA