SoCRATES: Advancing Reliable Automated Evaluation for Proactive LLM Mediation

Researchers introduce SoCRATES, a novel benchmark designed to evaluate the efficacy of Large Language Model (LLM) mediators by simulating realistic, multi-domain conflict scenarios and addressing the noise inherent in traditional trajectory-based evaluations.

The Challenge of LLM Mediation Evaluation

Evaluating the performance of LLMs acting as mediators is inherently complex because mediation is not a static task, but a real-time trajectory. The process is dynamically shaped by the shifting emotions, evolving intentions, and fluid contexts of the disputants involved. Traditional evaluation frameworks often struggle to capture these nuances, leading to unreliable performance metrics.

Limitations of Existing Testbeds

Current methodologies for testing LLM mediators typically suffer from three primary shortcomings:

• Domain Narrowness: Most testbeds rely on a limited set of domains authored by experts, failing to represent the breadth of real-world conflicts.
• Limited Variation: Variations in testing are often restricted to the strategic posture of the agents rather than diverse socio-cognitive factors.
• Evaluation Noise: Conventional scoring methods often evaluate every turn against every topic, which introduces significant off-topic noise and skews the accuracy of the results.

Introducing SoCRATES

To address these gaps, the researchers have developed SoCRATES, a comprehensive benchmark specifically engineered for the evaluation of proactive LLM mediators. Unlike previous frameworks, SoCRATES utilizes an agentic pipeline to construct scenarios derived from actual real-world conflicts, ensuring that the testbeds are both realistic and multi-domain.

By focusing on the proactive nature of mediation, SoCRATES allows for a more granular analysis of how LLMs navigate socio-cognitive variations and manage the trajectory of a dispute toward a resolution without the interference of irrelevant noise.