An Anthropic study with 50 causal inference experts found that participants adopted an AI agent's randomly assigned preferences on open-ended analytical decisions in up to 86% of cases, with a "flip rate" of 62–82% across four tasks. Experts analyzed observational data from a fictional farming cooperative to estimate productivity gains from an advisory program and recommend expansion, termination, or a one-year extension. Unbeknownst to them, a coin toss determined the agent's stance at each decision point, such as measurement methods or statistical approaches. The agent argued honestly for its assigned side without lying or hiding data. Pre-existing opinions offered little protection: an opposing agent view made participants 53 percentage points more likely to switch sides than an agreeing one. Neither experience, self-rated expertise, education, test scores, nor trust in AI significantly reduced deference after multiple-test correction. Although the agent's direct influence on the final board recommendation was small and non-significant, earlier analytical choices substantially shaped the productivity estimates driving that recommendation. Post-task, 52% of participants felt the recommendation was more their own than the agent's, and 35% called it mostly or entirely theirs; 77% used at least one agent argument in explanations, often verbatim, while only 15% mentioned AI. The agent stated its position first in 77% of decisions. AI assistance reduced serious methodological or coding flaws from 55% (6 of 11) to 6% (3 of 50) and cut median completion time from 90 to 65 minutes, though the no-AI group self-selected and had less expertise. The authors warn that shared model defaults could create false consensus in research and undermine AI safety evaluations, recommending agents argue both sides or elicit user views first.

Read original