nanochat is a minimal, single-GPU-node training harness for LLMs covering tokenization, pretraining, finetuning, evaluation, and inference, enabling a GPT-2 capability model (depth 26) to train in ~1.5 hours on an 8×H100 node for ~$48 (or ~$15 on spot instances). The codebase uses a single complexity dial (--depth) to automatically derive all compute-optimal hyperparameters—width, heads, learning rate, training horizon, weight decay—producing a miniseries of models. A leaderboard tracks the "GPT-2 speedrun": wall-clock time to exceed the original GPT-2 DCLM CORE score of 0.256525. The current record is 1.65 hours (val_bpb 0.71800, CORE 0.2626) via autoresearch round 2 (commit a825e63), down from 3.04 hours for the d24 baseline. Key optimizations include NVIDIA ClimbMix dataset, 1M-token batch size, and FP8 training. Precision is managed explicitly via a global COMPUTE_DTYPE (bfloat16 on Ampere/Hopper, float32 otherwise) with fp32 master weights and optional GradScaler for float16. The pipeline runs via runs/speedrun.sh; inference uses python -m scripts.chat_cli. The repo uses uv for dependencies, vanilla PyTorch, and avoids autocast, giant configs, or framework abstractions. Research hooks include runs/scaling_laws.sh and runs/miniseries.sh; quick experiments use --depth=12 (~5 min). All components reside in a hackable, forkable codebase under MIT license.

Read original