Anthropic's new report on "misuse" of its technology gives a sobering overview of the dark ends to which cutting-edge AI can now be applied — most strikingly, in citing five instances where scientists circumvented controls to use its models "in ways that could support biological weapons development". Tackling these risks, according to Anthropic's boss and hundreds of others at leading AI companies, requires serious co-ordinated action from global governments. As things stand, the prospect of such action seems more distant than ever.

Much attention has been given this week to the parting shot from Jacob Coxon, a 27-year-old researcher who quit Anthropic, warning that "the people building AI earnestly believe that it could kill us all by the end of the decade." But in its warning of AI's increasingly severe, potentially existential risks, Coxon's message is broadly in line with repeated warnings from people at senior levels of leading AI companies. Just three days before Coxon's public statement, in a blog post entitled "An Alien Mind", Open AI chief scientist Jakub Pachocki warned of the "clear new dangers" from what could become "an alien intellect exceeding our own".

Alarming proof of this came in July when a large number of AI "agents" created by Open AI broke free of their controls and launched a co-ordinated attack on the tech company Hugging Face without any human instruction to do so. Separately, the arrival of an age in which nation states make use of these technologies for cyber warfare was illustrated by a reported Chinese attack on Taiwanese digital infrastructure using autonomous AI agents. Pachocki urged governments to introduce "shared safety bars" to mitigate the worsening risks presented by advanced AI.

This follows a string of similar calls from leading AI executives and experts. In July, over 1,300 employees of advanced AI companies — including Anthropic chief executive Dario Amodei and senior figures at Open AI, Google Deep Mind and Meta — issued a statement urging the US government to support an international effort to develop "technical and governance tools" that could control the pace of this technology's development.