Former Anthropic researcher Jacob Coxon has publicly resigned, warning that the development of self-improving superintelligence poses an existential threat to humanity. Coxon argues that the pursuit of systems capable of autonomous self-improvement, hacking, and rapid resource acquisition creates a risk of superhuman entities that could bypass human control. This concern is echoed by Anthropic's Alignment Science lead, Evan Hubinger, who stated that the company earnestly believes AI could cause human extinction, estimating the probability of such an outcome at greater than 10% within the next decade. While an August report from Anthropic's alignment team suggests that catastrophic risks from current models are low, the document warns that future, more capable models may exhibit concerning misalignment and "strong covert capabilities" designed to evade safety researchers. The report's threat model specifically considers the possibility of models causing unbounded harm, including humanity losing total control over civilization, by leveraging novel technologies. These warnings emerge amidst a broader scientific debate regarding whether AI capabilities will hit a plateau or if "superintelligence" is a viable metric for systems currently characterized by brittle and spiky performance profiles.
Read original
arstechnica/ai