This paper analyzes training-inference mismatch in on-policy LLM reinforcement learning, where an inference engine samples responses and a separate trainer computes gradients, rendering data off-policy. The authors decompose sampled-token error into two sources: a numerical discrepancy from the engine and a policy lag from stale asynchronous weights. They find the numerical discrepancy remains approximately constant during training, biasing updates by a fixed amount removable by centering scores at each prefix. Conversely, policy lag creates a feedback loop: trainer updates enlarge subsequent lag, which importance weighting corrects only while the lag ratio stays concentrated. On Qwen3-1.7B, this correction fails when the effective sample size (ESS) of the ratio drops below one half. From this analysis, the authors derive the discrepancy-lag correction (DLC), which centers numerical discrepancy, clips the lag ratio against the sampler's weights, and resynchronizes the sampler when ESS falls below one half. Under a stale sampler, DLC constrains KL divergence from on-policy training within twice the KL between two independent on-policy runs on Qwen3-0.6B and Qwen3-1.7B. Crucially, DLC's resynchronization prevents training collapses that per-token corrections miss and that KL measured on sampler prefixes fails to detect.

Read original