Researchers introduce Denoising Token Credit Assignment (DTCA), a framework addressing token-level credit assignment challenges in reinforcement learning (RL) for diffusion language models (DLMs). Because DLMs generate text through iterative denoising, final outcome rewards poorly indicate how individual tokens contribute to the final output. DTCA estimates token contributions by measuring the improvement in prediction confidence of remaining masked positions after a token is committed, grouping these signals by part-of-speech (POS) categories. These contribution weights guide the selection of informative denoising timesteps and the assignment of token-level learning signals during RL training. DTCA further calibrates these signals based on token-level entropy changes induced by policy updates. Evaluated on LLaDA-8B-Instruct, DTCA consistently improves mathematical reasoning across varying generation budgets, achieving gains of up to 9.0 points on MATH500 and 5.3 points on GSM8K over the base model.

Read original