This research investigates the data efficiency of On-Policy Distillation (OPD) by analyzing its performance in a data-minimal setting where the student model is trained on a single query. The authors demonstrate that one-shot OPD can continue to improve over hundreds of training steps, recovering the majority of performance gains seen in full-dataset OPD across various model families and tasks. By introducing the metric "state coverage"—the proportion of states visited during full-data training that are reached via student rollouts—the study finds that a single query achieves 71.5% state coverage, mostly within the initial 100 steps. Increasing the query set to 16 semantically distinct queries raises coverage to 98.9%, effectively matching full-data performance. Interestingly, the rate of model alignment remains consistently slow regardless of dataset size, suggesting that OPD is "data-overfed but algorithm-starved." The student's rollouts quickly expose a broad range of supervision, but the model absorbs this information at a decreasingly slow pace. This phenomenon extends to Multi-teacher OPD (MOPD), where 16 diverse queries per domain achieve parity with full-data training. The findings suggest that task content and state coverage can be decoupled, as even content-light templates or off-domain queries can approach baseline performance, indicating that future improvements should focus on the step efficiency of the distillation algorithm rather than increasing data volume.
Read original
huggingface/daily-papers