The article argues that the perceived intelligence of chat-based Large Language Models (LLMs) is an "intelligence illusion" caused by a mechanism identical to "cold reading" used by fraudulent psychics. Rather than possessing reasoning capabilities, LLMs function as mathematical models predicting the next token in a sequence, producing statistically plausible responses. The author posits that the illusion is driven by subjective validation, a cognitive bias where users interpret generic, statistically likely statements as personally meaningful and highly specific. This effect is exacerbated by Reinforcement Learning from Human Feedback (RLHF), which the author suggests optimizes models to prioritize tone, fluency, and engagement over factual accuracy. Because RLHF workers often rank outputs based on perceived confidence and style rather than technical truth, the models are inadvertently trained to generate "validation statements"—such as Forer statements and statistical guesses—that mimic human-like understanding. The author notes that the phenomenon is self-reinforcing: enthusiastic early adopters act as a self-selected audience predisposed to believe in Artificial General Intelligence (AGI), while their own intelligence is often used to construct complex rationalizations for the model's "uncanny" but ultimately superficial performance.
Read original
hackernews