The authors benchmark ten self‑distillation techniques for agentic reinforcement learning against the GRPO baseline on four text‑based environments—ALFWorld, ScienceWorld, Search‑QA, and WebShop—using three different backbone model sizes. They conducted 132 independent training runs, accumulating 12,124.4 H800 GPU‑hours, and evaluated each configuration with eight rollout seeds for both the pretrained backbones and the eleven trained methods (the ten approaches plus GRPO). Beyond final task scores, they tracked learning curves, training stability, and measured computational overhead. Their analysis shows that improvements over GRPO are narrow and inconsistent: no single method surpasses GRPO across all tasks and model scales. Moreover, agents that achieve high final performance often exhibit volatile training dynamics, indicating that strong end results can conceal instability. Finally, increasing training expense does not reliably yield better outcomes, as higher cost does not consistently correlate with performance gains. By providing this extensive, reproducible benchmark, the study aims to clarify the true trade‑offs of self‑distillation methods and steer future work toward more reliable and cost‑effective agent training strategies.

Read original