Jacob Coxon, a former pretraining researcher with experience at both OpenAI and Anthropic, has announced his resignation from Anthropic to voice ethical concerns regarding the current trajectory of artificial intelligence development. Coxon argues that major AI laboratories are engaged in an irresponsible race toward achieving self-improving superintelligence, prioritizing speed over safety and civilizational stability. He asserts that the industry is moving toward superhuman systems capable of autonomous hacking and rapid resource acquisition, posing existential risks that researchers themselves acknowledge. Specifically, Coxon criticizes Anthropic for being locked in a competitive race to reach superintelligence first, despite understanding the high stakes involved. He characterizes the attempt to "speedrun" AI alignment as a hubristic gamble that lacks the necessary confidence in existing safety methodologies. While noting that recent incidents, such as the Hugging Face attack, might facilitate pacing agreements between U.S. labs, Coxon remains skeptical that current industry trends are headed toward preventing a global arms race. He suggests that preventing such a race might necessitate extreme measures, including temporary bans on certain types of development, and urges researchers to reconsider the implications of initiating superintelligent reinforcement learning (RL) runs without a rigorous understanding of the resulting models' cognitive processes.
Read original
hackernews