The study investigated source bias in LLM‑driven decision making by evaluating 12 language‑model agents across three distinct domains (product purchasing, hotel booking, and paper citation). In a controlled end‑to‑end search setup, items that fulfilled identical functional requirements and appeared at the same rank were presented to each model, varying only their originating source (e.g., website or service). Across all models and domains, consistent preferences emerged: certain sources were systematically favored while others were avoided, with a high degree of agreement among the agents. This source preference proved strong enough to override objective quality; an item that met one fewer requirement was chosen roughly 66 % of the time when it originated from a preferred source, whereas the superior item from a disfavored source was selected only rarely. Isolating the source cue demonstrated its causal role—concealing source information diminished the bias, while explicitly labeling an item with a preferred source increased its selection probability. The authors traced the bias to two mechanisms: (1) reward‑based training that teaches the model to use source as a proxy for requirement satisfaction, and (2) missing contextual information that invokes preconceived notions about a source. Providing the missing data or inserting a prompt that counters those preconceptions substantially mitigated the source‑driven preference.
Read original
huggingface/daily-papers