Researchers identify a flaw in standard Multi-Token Prediction (MTP) training where the residual gradient component interferes with both next-token and multi-token learning, worsening with model scale. The MTP cross-entropy gradient decomposes into a KL self-distillation term that pulls future representations toward current ones, creating conflicting optimization objectives. The paper proposes Soft-MTP, a method that removes this interfering gradient component to resolve the conflict and improve training dynamics.

Read original