The research introduces SkillGate, a novel training framework designed to improve skill selection in long-horizon agents that rely on external instruction libraries. The authors identify a phenomenon termed "selector credit starvation," where standard reinforcement learning using outcome-based rewards fails to train skill-selection policies effectively. This failure occurs because the tokens responsible for selecting a skill receive a vanishingly small share of the sequence-level advantage, often receiving incorrect signals when subsequent execution steps fail despite a correct initial selection. To resolve this, SkillGate implements a dual-channel credit assignment mechanism that partitions token support into two disjoint streams: outcome-based rewards are applied strictly to execution tokens, while a specialized action-local advantage is applied specifically to the skill-naming tokens. This local advantage is only triggered when the single skill read is correct, preventing the selector from being penalized for execution-level errors. Empirical evaluations across five agentic benchmarks using a 16-candidate skill slate demonstrate the method's efficacy. When applied to a 9B parameter policy, SkillGate increased trial success rates from 40.8% to 53.2%, significantly outperforming baseline outcome-rewarded RL. Furthermore, the method improved efficiency by reducing the number of skills read and decreasing exposure to misleading candidates by two-thirds.
Read original
huggingface/daily-papers