Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions

Researchers investigate the robustness of Large Language Models (LLMs) in simulating user-specific stances within online discussions, employing a counterfactual context revision framework to determine if model outputs reflect stable user beliefs or are overly sensitive to contextual shifts.

Overview of LLM-Based Social Simulation

The deployment of Large Language Models (LLMs) to simulate social media users has become a prevalent method for predicting human behavior and inferring how specific individuals might respond to evolving online discussions. By leveraging user data, these models attempt to emulate the "stance" or ideological position of a target user regarding a particular topic.

The Challenge of Stance Stability

A critical question in AI safety and social computing is whether these simulations capture genuine, user-specific beliefs or if they are merely reacting to the immediate conversational environment. If a simulation is highly sensitive to semantically independent changes in the context, the resulting "stance" may be an artifact of the prompt rather than a reflection of the simulated user's underlying persona.

Methodology: Counterfactual Context Revision

To audit these simulations, the authors introduce a framework based on counterfactual context revision. The process involves taking an original online conversation and introducing modifications to the context. By analyzing how the LLM's predicted stance shifts in response to these revisions, the researchers can evaluate the stability and reliability of the stance simulation.

Note: Due to the truncated nature of the source text, specific results, the exact nature of the "target" inference, and the final conclusions of the study are not available in this summary.