NVIDIA released Nemotron-Labs-Sigma, a family of 14 open checkpoints at 3B and 8B parameters implementing continuous diffusion language modeling in a learned 16-dimensional embedding space, diverging from discrete token diffusion used by LLaDA and Dream. The architecture employs a Ministral-style decoder-only Transformer with grouped-query attention, SwiGLU, and rotary embeddings, trained from an autoregressive warm start on ~1T tokens followed by ~0.3T tokens of diffusion training. Generation proceeds blockwise (default 64 tokens) with denoised blocks committed to a KV cache. Sigma-8B base achieves a 71.16 average on NVIDIA's 6-task suite (256 DDPM steps), surpassing NLD-8B (70.22), Dream-7B (58.74), and LLaDA-8B (41.78), with 65.12 on HumanEval and 65.74 on MBPP. Sigma-8B-instruct scores 91.96 on GSM8K and 78.40 on MATH-500 but only 17.00 on AIME25, yielding a 10-task instruct average of 59.14 versus NLD-8B's 59.92. Few-step distilled checkpoints (NFE 2, 4, 8, 16) retain performance; Sigma-distill-8B-nfe8 averages 70.34 on the base suite. Self-speculative variants use a diffusion head to draft 32-token blocks verified by an AR head, matching greedy AR output exactly. On a single GB200 with SGLang, the 8B speculative model achieves 3.46x average speedup over AR decoding (peak 4.20x), exceeding NLD-8B's 3.04x. Models require NVIDIA GPUs (Ampere through Blackwell) in bfloat16; base models generate 120-140 tokens/s on one H100. All checkpoints are available on Hugging Face under the NVIDIA Nemotron Open Model License for commercial use.

Read original