Researchers tested whether frontier LLMs learned protein sequence patterns by evaluating next-amino-acid prediction on 668 little-studied bacterial and archaeal proteins from 167 Pfam families of unknown function (no solved structures, ≤3 papers, no close UniProt relatives). Each protein was cut at five positions; models predicted the next residue given the prefix. A shuffled control destroyed sequence order while preserving composition. Claude Fable 5.2 achieved 54% accuracy on real proteins versus 9% on shuffled (45-point gap), outperforming all open protein language models including ProtGPT3-10B and ProGen3-3B. Opus 5.5 matched ProtGPT3-10B (34.9% vs 38.4% real; 7.5% vs 10.7% shuffled; identical ~27-point order-specific gaps). GPT-6 Astra, Gemini 4 Carbon, and seven open-weight models (DeepSeek V4.1 Flash, Qwen 3.8, Kimi K3.1 Pro, GLM-5.3, MiniMax M3, gpt-oss-120b, Nemotron 3 Ultra) scored near zero on both real and shuffled. Accuracy scaled with family size for Claude and protein LMs, plateauing for Fable 5.2 at ~300 sequences/family. Knowledge transferred to 56 proteins published post-training cutoff (August 2026) with no close prior relatives. DNA prediction was limited to coding sequences. Fable 5.2 also produced longer correct runs (88 full 10-residue continuations vs 12 for Opus 5.5, none for others). Results indicate Claude models were trained on protein sequences at scale, not merely protein literature.
Read original
hackernews