The article provides a reproducible guide for running NVIDIA's Dory 1.0 4B model using a pinned fork of llama.cpp (commit 98a74c9). The 3.67 GB Q5_K_M GGUF (SHA-256 96080be4...) is downloaded from a fixed Hugging Face revision. GGUF metadata reveals an architecture of 16 input layers, 40 recurrent layers, and 16 output layers (3.98B total parameters, 1.84B in the shared recurrent block) with a default loop count of 4, contradicting the model card's recommendation of 2. A cost model predicts 4 loops apply 2.51x the parameters per token and read 2.28x the weight bytes versus 1 loop. KV cache behavior is validated: "per_loop" mode allocates separate caches per recurrent layer per loop, growing from 306 MiB (1 loop) to 816 MiB (4 loops) at 4,096 context, while "last_loop" holds constant at 306 MiB. Decode speed on a T4 GPU slows 3.82x at 4 loops versus 1 loop (per_loop), closely tracking the 2.51x parameter ratio. Next-token distribution analysis on 50 GSM8K prompts shows "last_loop" matches "per_loop" exactly when prompts fit in one micro-batch (UBATCH=512), but diverges with 16-token chunks (mean TV > 0). A paired GSM8K evaluation (30 questions, seeded, reasoning on) measures accuracy, token counts, and wall time across five configurations; results are appended to JSONL files for resumability. Correctness gates confirm seeded sampling determinism and GPU/CPU top-1 token agreement.
Read original
marktechpost