This research investigates the mechanics of multi-turn long-horizon planning in foundation model agents through a controlled, unified environment. The authors analyze planning evolution across three distinct stages: acquisition, shaping, and integration. During pre-training, the study finds that constructing explicit world models via Chain-of-Thought (CoT) state transition modeling significantly enhances long-horizon generalization. Notably, while atomic skills are insufficient for compositional generalization, minimal long-horizon data is effective, whereas suboptimal trajectories cause performance degradation due to error amplification. For the shaping stage, the researchers utilize Group Relative Policy Optimization (GRPO) and On-Policy Agentic Distillation (OPD) to distinguish general planning patterns from task-specific knowledge. Results indicate that OPD maintains a broader effective application region than GRPO, particularly in low-quality or long-horizon scenarios, by providing more consistent update directions. Finally, the study introduces Multi-Teacher On-Policy Distillation (MOPD) to facilitate capability integration. The findings demonstrate that MOPD allows agents to converge toward shared planning patterns across different environments. The research concludes that compatible patterns foster cross-environment generalization and support continual learning, while conflicting patterns lead to severe interference, providing a roadmap for optimizing agentic post-training through systematic distillation.
Read original
huggingface/daily-papers