Vaibhaav Tiwari implemented the DeepSeek-V3 architecture in 190 lines of pure JAX without Flax or Equinox, training a 36.2M parameter model (13.6M active per token) from scratch on a MacBook Air M5 (24 GB). The implementation replicates key architectural components: Multi-head Latent Attention (MLA) compresses KV caches to 144 numbers per token per layer versus 480 for standard attention; Decoupled RoPE uses one rotary key shared across heads kept outside the compressed latent; DeepSeekMoE employs one shared expert plus 16 routed experts with top-4 selection per token; auxiliary-loss-free balancing applies per-expert biases to nudge routing toward idle experts without modifying weights; a sequence-wise balance loss (α=1e-4) provides complementary regularization; and Multi-token Prediction (MTP) adds a module predicting the next-next token weighted at λ=0.3 in the loss. Training used bf16 compute with f32 master weights on 21.7M tokens from 207 public-domain detective novels (Hammett repeated 3×) over 12,000 steps (~98M tokens, ~4.5 epochs) in ~2.6 hours at ~10.7k tokens/s. Validation loss dropped from 5.02 to 3.51. Expert load with balancing bias ranged 5.7–6.9% per expert (ideal 6.25%) versus 0.09–18.6% without bias at step 2000; validation loss was nearly identical (3.921 vs 3.923), confirming the bias evens expert utilization without loss penalty. The code uses dense MoE compute (all experts run, unpicked zeroed), explicit MLA latent decompression, and omits FP8 training, DualPipe, node-limited routing, YaRN, and post-training. Checkpoint and data hashes are provided for reproducibility.
Read original
hackernews