A 2026 study evaluated 26 language models on 6,438 test instances spanning input lengths from 8 K to 1 million tokens using the ATLAS framework, which measures performance across eight capability dimensions and separates foundational retrieval from complex application reasoning. ATLAS aggregates scores with a harmonic‑mean ATLAScore to penalize imbalanced profiles and propagates uncertainty in the final aggregate. Results showed that Gemini 3.1 Pro Preview achieved the highest score for contexts up to 128 K tokens, whereas Claude Opus 4.6 topped the ranking at the full 1 million‑token scale; seven models shifted at least two positions when the evaluation range was extended from 8 K–128 K to 8 K–1 M. A complementary CorpusQA benchmark reported that Gemini 2.5 Pro’s overall accuracy fell from 80.19 % at 128 K tokens to 50.73 % at 1 M tokens—a 29.46‑point decline—while standard retrieval‑augmented generation managed only 17.38 % accuracy at 1 M tokens compared with 44.95 % for a Memory Agent approach using the same model. These findings demonstrate that model effectiveness varies with context length and that simply expanding the context window does not guarantee reliable reasoning over long inputs.
Read original
dev.to