Diffusion language models offer an alternative to autoregressive models by generating entire sequences simultaneously through iterative refinement rather than left-to-right token prediction. A foundational approach is Masked Diffusion Language Modeling (MDLM), which treats text generation as an unmasking process. In this framework, a forward process applies a masking schedule ($\alpha_t$) to transform clean text into a fully masked sequence, while a reverse process—typically a bidirectional transformer—is trained to predict missing tokens. While standard MDLMs are limited to fixed-length sequences, advanced implementations use block diffusion to support variable lengths and KV caching, or encoder-decoder architectures to accelerate inference. To enable error correction, techniques like remasking or Uniform State Diffusion (UDLM) allow the model to revise previously generated tokens. Furthermore, classifier-free guidance (CFG) enables high controllability by interpolating between conditional and unconditional distributions. These advancements have led to significant performance gains, with models like Mercury 2 achieving speeds of 5–10× faster than autoregressive models on standard GPUs, and NVIDIA's Nemotron reporting 2–8× throughput improvements. While autoregressive models remain the standard, diffusion models demonstrate superior potential for parallel generation, property-targeted scientific design (e.g., ESM3 for proteins), and inference-time scaling.

Read original