llama-modchip enables running the 284.33B-parameter DeepSeek-V4-Flash MoE model (13.17B active across 43 MLA layers and 256 routed experts) entirely in 64 GB system RAM on x86_64 and ARM64 CPUs without GPUs. By transcoding the base UD-IQ1_S checkpoint (76.87 GiB) into a custom 56.56 GiB Fractal-64 GGUF format (1.71 bpw), the project fits the model plus a 256K context KV cache into 57.61–58.9 GiB VmRSS, eliminating NVMe swap on Linux. The Fractal-64 scheme preserves embeddings, dense layer 0, 43 shared experts, and MLA projections in Q4_K/Q5_K/F32; transcodes MLA wq_b and attention outputs to block-64 BF16-scale Q2_0 (3.75 bpw); and quantizes 256 routed experts primarily to IQ1_S (1.5625 bpw) with analytical second-order SwiGLU-H2 scale matching, retaining sensitive boundary layers in IQ2_XXS (2.06 bpw) and IQ1_M (1.75 bpw). CPU kernel patches deliver a 77% prefill speedup (25.0 → 44.36 tok/s on 16-core Zen 5) via a four-level ZMM AVX-512BW + L1D/L2/L3 multi-column tiling schedule ({8,4,3,2}), register-stationary MQA8 FlashAttention, and non-temporal _MM_HINT_NTA MoE streaming to protect hot KV cache lines. Self-speculative drafting pairs a 1.64 MB L2/L3 Zobrist n-gram drafter (Stockfish-PVS v2) with /dev/shm KV slot snapshots restoring 40-tool Claude Code contexts in 13.1–32.6 ms. Default Top-6 expert evaluation maintains 1.00000 logit cosine similarity and 0.0000 KL-divergence versus unpatched execution; optional Top-5/Top-4 modes trade minor perplexity deltas for 9.87–10.74 tok/s decode throughput. The Nemotron-3.5-30B-A3B Fractal-64 variant (3.75 bpw, 14.35 GiB) achieves 25.0–60.4 tok/s on 16–32 GB RAM and 111.6 tok/s on 8-channel Threadripper. Pre-compiled zero-dependency binaries and a unified ./modchip CLI handle quantization, benchmarking, and agentic workflows across Linux, Windows, and macOS.
Read original
hackernews