This research investigates length inflation during on-policy distillation (OPD), a phenomenon where student models generate excessively long sequences that may exhaust generation budgets. The authors identify "termination-token mismatch" as a primary driver of this behavior. Through experiments involving Qwen3, Llama, and Gemma model families, the study reveals that even when base students and post-trained teachers share identical declared stopping sets, they often assign disparate stopping probabilities to different EOS tokens. This discrepancy can suppress the student's natural termination preference without successfully adopting the teacher's preferred stopping action. The researchers demonstrate that merely aligning decoding stopping sets is insufficient to resolve the issue. Instead, treating functionally equivalent EOS tokens as a shared semantic stopping action significantly mitigates mismatch-induced length inflation across all tested model families. Furthermore, a stage-wise analysis using K2-Horizon training reveals that termination preferences undergo substantial shifts during the distillation process. While the proposed termination-handling corrections address much of the issue, the authors note a distinct form of length inflation that emerges late in the OPD run, persisting even after termination alignment is achieved. This suggests that while termination mismatch is a critical factor in OPD length dynamics, it is not the sole cause of sequence elongation.
Read original
huggingface/daily-papers