Zhipeng Chen, Wayne Xin Zhao, and Ji‑Rong Wen introduced Ares (Adaptive Rank‑Effective Spectral Optimization), an optimizer that dynamically adjusts the rank of gradient updates based on the spectral energy distribution of LLM gradients during reinforcement learning with verifiable rewards (RLVR). Analyzing gradient spectra across training, they observed that in RLVR the energy concentrates into a progressively smaller effective subspace, a pattern absent in supervised fine‑tuning and when rewards are random, and that the rank required to retain a fixed proportion of energy steadily declines as training proceeds. Leveraging this observation, Ares estimates the instantaneous effective rank, monitors its evolution, and restricts each update to the corresponding low‑rank subspace, thereby aligning the optimizer’s geometry with the training dynamics. Experiments on LLMs ranging from 1 B to 65 B parameters show that Ares consistently outperforms Adam, Adafactor, and LAMB in both convergence speed and final accuracy across multiple RLVR algorithms and benchmark tasks, with relative gains up to 7 % in win‑rate on the Open‑Domain QA suite. The authors release the implementation and evaluation scripts at the provided URL.

Read original