Researchers have introduced Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO) to address inefficiencies in multi-objective reinforcement learning for language models. Traditional methods typically scalarize reward vectors using fixed weighted sums before group-wise standardization, which causes two issues: distinct reward profiles receive identical advantages, and optimization effort is wasted on already-saturated objectives. To solve this, SA-MRPO independently standardizes each reward objective and applies adaptive discounting based on a batch-level estimate of objective saturation. This mechanism dynamically reallocates gradient budget toward under-optimized goals, even reversing the sign of updates to prevent over-optimization of satisfied objectives. Experimental results demonstrate the efficacy of this approach across various tasks. In mathematical reasoning involving two or three reward objectives, SA-MRPO outperformed Group-relative Direct Preference Optimization (GDPO) on the AIME24 benchmark with gains up to 5%. On adaptive reasoning tasks, the method achieved an average accuracy improvement of 3.8%, with a peak increase of 9.2% on AMC23. Additionally, coding benchmarks saw pass rate improvements of up to 2.3%. Throughout these tests, SA-MRPO successfully prioritized harder objectives while maintaining performance on easier, already-mastered goals.
Read original
huggingface/daily-papers