To address the issue of exposure bias in diffusion models—caused by error accumulation and frequency-dependent SNR discrepancies during iterative sampling—the researchers propose Spectral Alignment (SPA). The authors identify that mismatches between training and inference distributions vary across different models and timesteps, rendering fixed correction methods ineffective. SPA functions as a lightweight, guidance-based calibration technique that aligns the power spectrum of intermediate model predictions with a pre-computed prior. The methodology involves a two-stage process: first, an offline phase where a parametric spectrum model is fitted using training data, and second, an inference-time phase utilizing efficient Fast Fourier Transform (FFT)-based gradient computation to guide the sampling process. Experimental results show that SPA is highly versatile, providing consistent improvements across various architectures, including pixel-space models like DDPM and ADM, latent diffusion models such as SD2.0 and SDXL, and flow-matching models like SD3.5 and FLUX. The method is designed to be complementary to Classifier-Free Guidance (CFG) and introduces negligible computational overhead, estimated at only 3-4%. By correcting spectral discrepancies, SPA mitigates the systematic errors that typically degrade image quality during the generative sampling process.

Read original