The DRACO (Distributing Rubric-based Advantage for Credit Optimization) framework addresses the challenge of credit assignment in long-horizon agent training within outcome-blind settings where programmatic verifiers are unavailable. Unlike traditional methods that provide a single scalar reward per trajectory—which often results in a sparse and noisy signal—DRACO utilizes dynamic multi-criteria rubrics that evolve alongside the policy's capabilities. The method scores these rubrics once per completed trajectory and then employs a closed-form redistribution mechanism to allocate the judgment across the specific steps responsible for the annotated rubrics. This process generates differentiated per-step advantages within the Group Relative Policy Optimization (GRPO) framework without requiring a trained attribution module. Empirical evaluations demonstrate the effectiveness of this approach: on the AppWorld benchmark, DRACO achieved a 15.9-point improvement over the base model and outperformed GRPO trained with sparse ground-truth rewards by 5.3 points. Furthermore, in out-of-domain testing on Tau-Bench, the method yielded a 5.3-point gain over the base model, surpassing both ground-truth-reward training and existing rubric-based training configurations, even in the absence of a frontier judge.
Read original
huggingface/daily-papers