To mitigate the performance degradation caused by On-Policy Self-Distillation (OPSD) in complex reasoning tasks, researchers have introduced Negative Self-Distillation (NSD), a novel optimization framework. While OPSD often suppresses essential self-corrective behaviors and uncertainty by forcing models to imitate confident, privileged reasoning traces, NSD adopts a divergence-based approach. Rather than imitating ground-truth solutions, the model generates question-specific negative conditions—such as acting as a "careless reasoner"—and optimizes the student distribution to move away from these flawed reasoning paths. To prevent the catastrophic degradation of foundational language capabilities that occurs when unlearning objectives indiscriminately penalize both reasoning and linguistic tokens, the authors implemented a dynamic gating mechanism. This mechanism identifies and isolates reasoning-critical tokens, ensuring that gradient updates specifically target behavioral flaws while preserving the model's existing linguistic priors. Empirical results demonstrate that NSD consistently outperforms OPSD and various label-free, self-bootstrapping reinforcement learning (RL) baselines, proving more effective at enhancing reasoning capabilities through the avoidance of self-generated errors rather than direct imitation of privileged information.
Read original
huggingface/daily-papers