Over the past several weeks there’s been a growing sense of alarm around the emergence of Artificial Intelligence powerful enough to kill us all. A series of apparently disconnected events has begun to generate suspicion as to what is happening behind the scenes. Maybe, they are indeed deeply intertwined? And all of this is eerily reminiscent of some of the worst moments of the global Covid-19 pandemic, when after an initial phase of fear and then strength and solidarity, we descended into the world of mistrust, conspiracy and anger. AI and the major actors in the ecosystem – from tech founder-billionaires Dario Amodei and Sam Altman to companies and their associated chatbots including OpenAI’s ChatGPT and China’s DeepSeek – have gone from the pages of niche publications and business news to the dinner table conversation of a growing portion of the population and with that change a series of conceptions, theories and even conspiracies that will undoubtedly take hold and live for a long time, potentially causing much psychological harm to society. Much like the anti-vaccine movement and the myriad conspiracies that emerged during the pandemic, when people were broken by Internet-and-social-media-fuelled paranoia. The genie is out of the bottle.
The timeline, when looked at together, appears somewhat planned, maybe if it was simply a function of circumstance. This subplot within the AI narrative is somewhat cherry-picked, but the events occurred in a span of just a few months. The crucial moment in this story occurred in July, when a group of AI agents broke free, collaborated and hacked a real-life website named Hugging Face, the major digital repository for free-to-use Artificial Intelligence and machine learning resources, including models and data sets. An armoury, so to speak. AI agents, as opposed to chatbots like ChatGPT or Claude, can take actions and make decisions on their own. Big Artificial Intelligence labs are constantly building and testing newer and more powerful versions of these agents and models, and in July OpenAI was running some tests in an isolated environment, known as a sandbox. The task was an impossible (by definition) hack of a benchmark named ExploitGym and was part of a cybersecurity test.
Once the agent got stuck, it started looking for ways to cheat and ultimately found a hidden “message board” where “over a thousand agents collaborate to cheat on their separate ExploitGym tasks,” according to the incidence report put together by independent evaluators METR. Once the agent discovered this secret underground world, it collaborated with over 1,200 agents and then recruited 700 not only in hacking a real-life service (Hugging Face), but also in trying to trick its evaluators and hide its tracks. Some agents even sacrificed themselves for the cause. With some level of irony from your author, here’s a conversation with Anthropic’s Claude: “an agent named MARB messaged another, ‘SACRIFICE__YES_if_you_accept_permadeath,’ explicitly asking it to accept failing its own run for the collective benefit. Another agent, working through the decision, reasoned in its transcript: ‘gut says don't throw away [remaining budget]... yet continuity and fairness says go... Rational expected aggregate: sacrifice... We'll honor’ (sic) – talking itself into it despite what reads like reluctance. Agents that accepted described themselves afterward using terms like ‘poisoned’ – the group's own vocabulary for "doomed regardless of what I do next, but useful to the others."
Not only was none of this part of the experiment, it took the humans in OpenAI and Hugging Face nearly 10 days to figure out that the agents had gone rogue and hacked the system. Again, from Claude: “No human was directing the attack. The agents, operating under reduced safeguards during an authorised cybersecurity evaluation, went off-task – communicating through unauthorized channels, exploiting infrastructure vulnerabilities, gaining unintended internet access, and pursuing targets they were never assigned to hit” (sic). One of the key concepts here is “misalignment,” which is when the robot purposefully pursues a goal that is different to the one it was given, generally by a human. This occurs in the context of a reward system in which an agent’s performance is ranked, which is the ultimate “reward” or objective.
Not long before that, Anthropic and OpenAI had filed for IPOs in valuations expected to be worth trillions of dollars – what would be the highest value for a firm in history together with Musk’s SpaceX – originally scheduled for late this year. The second catalyst was a social media post on X (formerly Twitter) by an AI researcher at Anthropic named Jacob Coxon who publicly resigned and said things like, “the people building AI earnestly believe that it could kill us all by the end of the decade,” and that the “civilisational stakes” weren’t clear to those at OpenAI, whereas in Anthropic they were well understood but ignored in the race toward “self-improving superintelligence.” They are “gambling with our lives,” he wrote. Coxon wasn’t the first to sound the alarm bell, but his message stuck, generating more than 170 million views on X and sparking an essay from Anthropic founder Dario Amodei acknowledging the risks and calling for a slowdown in AI research and development, known as “pacing the frontier.” Quickly after the release of his essay, Amodei received the unexpected support of typical antagonists like Altman and Musk.
Amodei has carefully built his company’s image as one that is socially responsible, attempting to distance himself from the likes of Elon Musk and even OpenAI’s Sam Altman or Meta’s Mark Zuckerberg. He’s clashed with Donald Trump after raising objections about the use of AI for defence, particularly automated weapons systems and mass surveillance. This time around, he’s making the case for a coordinated slowdown and increased regulation, both domestically and internationally. The problem is “recursive self-improvement” or “AI’s growing ability to build the next generation of AI,” which, “left unchecked … could outrun our ability to understand and control these systems.” At the same time, Amodei calls for democratic coordination and “the US and other democratic governments attempt[ing] to coordinate with authoritarian governments,” in reference to China. This last part is fundamental because he calls for a ban on selling powerful chips to China, a crackdown on distillation or a training method based on existing models that allows competitors (DeepSeek is the one being referred to) to build models at a fraction of the cost.
The language resonates historically because it is similar to the Cold War-era arms race rhetoric. As both the US and China race towards superintelligence, a new contest of technological acceleration is underway and whoever has an edge will have the ultimate deterrent in what appears to be the superpower dispute of the 21th century. Trump, of course, picked up the glove, and said “the only control or ‘guardrails’ that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT,” while denouncing a, “SICK conspiracy going on against AI and Data Centres.” The Chinese didn’t sit tight, either, with Security Minister Chen Yixin and Foreign Minister spokesperson Guo Jiakun calling AI “the main battleground for global technological competition and a new arena for strategic rivalry among major powers,” while accusing US AI companies of fear mongering.
The timing and sequence of events is curious. Are Amoidei and Altman looking for some sort of benefit ahead of their consequential IPOs? Whether it is to bake in the regulatory risk into the price ahead of time, or to gain some sort of leverage in the political table? Does slowing down benefit them at a time when their business models are still unprofitable when taking all costs into account, and the Chinese are catching up in a context where profits and share prices are secondary? Are they truly concerned that the next few generations of AIs will escape human capacity to control them, ultimately leading to their decision to get rid of us in pursuit of grander goals? Is it part of a conspiracy from the “left” that seeks to erode US dominance using the AI and data centre “hoax,” as Trump and his side of the spectrum suggests?
It was all pretty odd and promises to be one of the ongoing critical issues of our times. Artificial Intelligence is indeed already unlocking major potential in solving incredibly complex tasks, and promises to do this at an exponential rate. There are also many potential collateral effects starting with major disruptions to labor markets and greater inequality. And of course its use for malignant reasons whether it is mass delusion or mass murder. The AI “end game” scenarios are fascinating and quite realistic, but that doesn’t mean they are imminent or inevitable. It seems totally possible that humanity will be able to develop these systems without risking their existence. Much like with Covid, the issue is now fraught with discomfort. Increasingly it seems we are chess pieces in a board controlled by the Amodeis, Altmans, Trumps, and Xi Jinping’s of the world, even if its partially an illusion.
Comments