TurboFieldfare is an open-source inference engine written in Swift and Metal designed to run the Gemma 4 26B-A4B model on Apple Silicon Macs with as little as 8 GB of RAM. Unlike standard wrappers, this custom runtime utilizes an SSD-backed streaming approach to manage the 14.3 GB model. It maintains only the 1.35 GB shared core and an FP16 KV cache in memory, while streaming required experts from the SSD on demand. The architecture employs an LFU cache for routed experts and uses chunked prefill (up to 128 tokens) to improve efficiency. Technical performance benchmarks show a decode rate of 5.1–6.3 tok/s on an 8 GB M2 MacBook Air, while an M5 Pro with 24 GB of RAM achieved 31–35 tok/s. The engine supports 4-bit MLX affine quantization for weights and an 8-bit router. Additionally, an optional 1.1 GB vision tower allows for image input on M2 or newer chips. The project provides a native macOS app, a CLI, and an OpenAI-compatible loopback server.

Read original