Wu, Han, and Cai propose Weakening-Induced On-Policy Self-Distillation (WISD), a framework that improves large reasoning models without external teachers or privileged information by contrasting the original model with a deliberately degraded version. They instantiate the weaker counterpart through two compression techniques: low-bit quantization and magnitude pruning. The on-policy distillation signal derives from the performance gap between the strong and weak models on unlabeled prompts. Evaluated across diverse architectures and scales on mathematical and coding benchmarks, both methods yield consistent gains. Quantization improves average accuracy by 4.0 percentage points over the baseline, while pruning achieves a 5.5-point lift, reaching state-of-the-art results without privileged data. Specifically, pruning elevates Qwen3.5-4B from 57.7% to 70.5% on HMMT 2025 and from 39.1% to 56.7% on LiveCodeBench v6. Quantization raises Qwen3.6-35B-A3B from 68.5% to 74.6% on HMMT 2025 and from 61.8% to 69.5% on LiveCodeBench v6. These results establish WISD as an effective self-distillation paradigm that leverages model weakening as a supervisory mechanism.

Read original