Ricursive Intelligence released Ripple-65B Preview, a 65-billion-parameter dense Transformer model pretrained from scratch on 12 trillion tokens of English text and code with an 8K sequence length, extending context to 1 million tokens via midtraining at 2M length. The architecture uses a thin, deep design (d_model 5120, 80 layers) with Gemma 3N-inspired stability features: overprovisioned attention heads, sliding-window attention, gated attention, global NoPE, value residuals, and FFN activation clipping. Training achieved 40% MFU using simple sharding without custom kernels. The $32M R&D budget covered ladder experiments from 4B scale, where a 16x code compute-efficiency gain was reported. Post-training combined midtraining, long-context SFT, and mixed RL with truncated importance sampling, score centering, and a linear length penalty, reaching 80.5 on Terminal-Bench 2.1 using only 1.4M rollouts—far fewer than Beam’s 100M+. On Terminal-Bench 2.1 at 128K context, Ripple held 80.5 while Beam dropped to 70.8. It also leads on SWE-bench Pro v2.0 Hard (79.1) and scores 98.3 on AIME 2025 and 90.5 on HMMT Feb 2026. Weaknesses include SciCode (39.4), DeepSWE 1.1 (44.3), and GPQA Diamond (85.2), trailing larger MoE rivals (501B–975B total params). The Apache 2.0 release includes BF16 and experimental NVFP4 checkpoints (MLP quantized, attention in BF16) targeting NVIDIA DGX Spark; the BF16 model runs at 250 tok/s on 2x B200 GPUs.
Read original
marktechpost