ContextRL: Enhancing Long-Horizon Reasoning in Agentic and Multimodal LLMs via Context-Aware Reinforcement Learning

Researchers introduce ContextRL, a novel reinforcement learning framework designed to mitigate the "needle-in-a-haystack" failure mode in Large Language Models (LLMs), improving their ability to isolate critical evidence within complex tool traces and multimodal inputs.

The Challenge of Evidence Retrieval in Complex Contexts

A persistent limitation in current Large Language Models (LLMs) is the tendency to overlook decisive pieces of evidence when they are embedded within extensive or complex contexts. This failure is particularly evident in agentic workflows—where a model must parse long tool execution traces—and multimodal tasks, where a subtle visual detail may be the key to a correct answer. When the critical information is a "small but decisive" element, standard supervision often fails to guide the model toward the correct reasoning path.

Introducing ContextRL

To address these shortcomings, Peiyang Xu et al. propose ContextRL, a context-aware reinforcement learning method. Unlike traditional RL approaches that focus primarily on supervising the final output (the answer), ContextRL introduces an indirect auxiliary objective to refine how the model interacts with its input context.

Mechanism and Methodology

The core innovation of ContextRL lies in its training objective. Instead of rewarding only the correctness of the final response, the method presents the model with a triad consisting of a query, an answer, and the associated context. By implementing this auxiliary objective, the framework encourages the model to develop a more acute sensitivity to the specific segments of the context that are most relevant to the solution, thereby enhancing long-horizon reasoning capabilities.

Applications in Agentic and Multimodal Systems

The authors highlight the effectiveness of this approach in two primary domains:

• Agentic LLMs: Improving the ability to identify critical lines within tool traces to ensure precise action execution and reasoning.
• Multimodal LLMs: Enhancing the detection of subtle visual cues within images that are essential for accurate multimodal reasoning.