The paper studies off‑policy reinforcement learning with verifiable rewards (RLVR) where data arise from asynchronous generation, rollout‑training mismatch, and response reuse. It shows that token‑wise importance weighting, which avoids multiplying likelihood ratios across an entire response, can yield unbiased policy gradients when full future KL credit is used, provided coverage and local expressiveness conditions hold. Under these assumptions the exact proximal optimizer becomes the unique stationary point of the update, even for mixtures of older policies, without requiring full‑response weights or a fitted critic. The authors prove convergence of continuous‑time tabular population updates under persistent coverage and time‑varying behavior policies, and derive a finite‑sample reward guarantee for linear logistic models given extra assumptions. Motivated by these results they introduce Proximal KL, a practical off‑policy RLVR algorithm that decouples the proximal reference from the sampling policy and replaces PPO clipping with bounded future KL credit. Experiments on Qwen3‑1.7B and Qwen3‑4B across seven off‑policy settings (two learning rates, response reuse, rollout staleness) show Proximal KL improves micro‑averaged accuracy on eight math benchmarks over GRPO by up to 2.58 %. On‑policy performance matches GRPO. Periodic‑refresh tests reveal Proximal KL avoids the collapses of PG and CISPO, attains higher training accuracy than GRPO and reaches it in fewer updates. Ablations confirm token‑wise correction outperforms sequence‑level and unweighted baselines, and that a finite future horizon improves AIME24 mean@16 and pass@16 while reducing correction variance via truncation.