Researchers have introduced Fuse, a novel multi-agent simulation framework designed to evaluate the social reasoning capabilities of Large Language Models (LLMs) in user-mediated consultation settings. Addressing the difficulty of verifying subjective social properties like human intentions, Fuse utilizes a structured setup where a target agent with a hidden motive interacts with other agents, including a user proxy. The user then consults an LLM assistant to infer the target's underlying motive, allowing the researchers to establish verifiable ground truth by construction. The framework's simulation faithfulness was validated via a human study involving 24,000 annotations. Testing 12 different LLMs, the study revealed that user mediation increases the complexity of social reasoning tasks. Key findings indicate that models are systematically sensitive to biased framing provided by the user, often require more granular details than humans to reach correct predictions, and do not necessarily benefit from longer conversational contexts, even when clarifying questions are possible. The authors have released both the Fuse framework and a dataset containing 21,000 examples to facilitate further research into how AI assistants navigate complex, subjective social narratives.
Read original
huggingface/daily-papers