Researchers have introduced an exploration-guided prompt scaffolding framework designed to optimize the reinforcement learning (RL) post-training of multimodal large language models (MLLMs). The authors identify a fundamental inefficiency in standard RL training, where training prompts are treated uniformly regardless of whether they are already saturated or too difficult to provide meaningful learning signals. To address this, they propose a method to dynamically adapt the prompt distribution using an Exploration Potential Score (EPS). This lightweight, rollout-based proxy utilizes on-policy rollout statistics and KL-regularized policy improvement theory to estimate prompt utility without incurring additional computational overhead. Instead of discarding low-utility prompts, the framework employs a teacher model to perform scaffolded rewrites that maintain the original task intent while increasing the informativeness of the training signal. This approach shifts the role of teacher supervision from output imitation to training-data refinement. When integrated with Group Relative Policy Optimization (GRPO) and tested on benchmarks like Geo3K and MMK12, the method demonstrated significant performance gains. Specifically, the framework achieved relative in-domain improvements of up to 9.7% and showed robust generalization on out-of-distribution datasets, including an 11.5% increase on MathVision and an 11.1% improvement on MMMU-Pro.

Read original