Stanford researchers Owen Queen and James Zou evaluated 50 widely cited claims about large language models against 20 models released between July 2024 and September 2026 by eight developers, finding that 44 percent of claims had expired, with an average lifespan of 15.7 months for the 22 findings with estimated expiration dates. After one year, 16 percent of claims expired; after two years, 39 percent. Claims about model limitations expire fastest: five of nine reasoning-related findings, including failures at counting letters in "strawberry," arithmetic in unfamiliar bases, lost-in-the-middle context effects, and sycophantic reversal under user pressure, are now expired. The researchers attribute this to clear correct targets for training. Conversely, claims about preferences persist: models still prefer their own outputs as evaluators, rate accepted and rejected papers similarly, exhibit linguistic spin susceptibility in medical abstracts, show sycophancy, left-leaning political bias, near-identical open-ended responses, and non-uniform random number generation. In a citation fabrication case study, GPT-3.5's 55 percent hallucination rate dropped to 0 percent for GPT-6.1 Sol and 1 percent for Claude Sonnet 5.5, yet Qwen3.7-Flash still fabricated 69 percent. On two-hop reasoning, Claude Sonnet 5.5 fails 10 percent of cases despite correct sub-answers. The Reversal Curse and time-question contradictions have weakened but persist. Social biases are mixed: left-leaning bias remains, gender stereotypes have weakened, and bias against African American English has flipped direction. Model-to-model variance is wide within the same generation; expired findings range from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol, uncorrelated with release date, provider, or size. The authors argue empirical findings must be framed as observations of specific models at specific times, with full documentation of model version, prompts, and code for reproducibility.

Read original