The smolmaxxing project demonstrates running large language models on severely memory-constrained devices — specifically a Xiaomi TV Box S (Amlogic S7D, 4× Cortex-A55, ~1 GB usable RAM) and a jailbroken iPad Air 2013 (Apple A7, ~700 MB usable RAM) — by patching llama.cpp with custom kernels and memory-management techniques. On the TV box, a 1-bit lookup-table kernel and repacked 2-bit/ternary formats optimized for Cortex-A55 enable Qwen3.5-2B at 7.7 tokens/s and the 16B mixture-of-experts model Ling-mini-2.0 at 5.1 tokens/s; the iPad achieves 24–26 tokens/s with 1-bit Bonsai-1.7B and 2.6–4.5 tokens/s with Ling-mini-2.0. Both devices run 8B and 16B MoE models within ~1 GB RAM by keeping only active experts resident and paging others from flash storage on demand with prefetching, while a vocabulary-pruning "headless" mode on the TV box frees additional memory by dropping rarely used output rows and stopping system services. For dense models exceeding RAM, weights stream from flash at full device bandwidth. All techniques — chip-specific NEON kernels, expert paging, flash streaming, and vocabulary reduction — are implemented as llama.cpp patches with measurable gains documented in docs/how-it-works.md; prebuilt GGUF models require the patched runtime. The repository includes device-specific setup guides, model rebuild tooling, benchmarks, and a full experiment log across 160 runs.
Read original
hackernews