Anthropic researcher, Jacob Coxon, recently announced he was leaving Anthropic because there’s a non-trivial chance AI is going to wipe us off the face of the earth. His claim was backed by his colleague Evan Hubinger, head of alignment, who agreed on Twitter/X saying many inside the company believe there is a >10% chance AI will kill all humanity in the next decade.
There’s a particular hubris in claiming your team is uniquely equipped to steer humanity safely into the AI age while, in the same breath, conceding it might kill us all anyway. Setting that aside, along with the obvious irony of a company called Anthropic building the very monster that may or may not destroy us, we can rest assured they are trying their best.
Hype literacy means knowing that claims like “AI will kill us all” are a function of the aggressive marketing dynamic inherent in the hype cycle – regardless of the narrator’s intent. The average investor will see this and say “I’d better invest in (insert frontier AI lab) because their product is so godlike that they may even do away with these tedious humans with their incessant need for coffee breaks and maternity leave.” It is not to suggest that this researcher is not being forthright, but the narrative only serves the incumbent. There’s even a term for it: “criti-hype”, coined in a blog post by Lee Vinsel in 2021. In short, the manufactured panic around emerging tech betrays a kind of “wishful worry”. The critic borrows the industry claim – the tech is unbelievably transformative – and accepts it outright. Only the framing is flipped: it’s not good, it’s bad. Both the detractor and the booster agree on the premise. That they disagree on the outcome doesn’t matter since the investor only hears “unprecedented, disruptive, transformative”. The negative reading complements the pitch.
For the rest of us NPCs, watching former lab employees catch a case of late-onset moral compass is growing tiresome. These researchers and programmers felt comfortable enough to plug away for years, building models that strip-mine our collective IP as colossal, energy-intensive data centres are erected in economically depressed areas. All with the goal of being the first ones to reach AGI (another vague marketing term) before the evil, non-aligned guys get there first. Once it becomes too rich for their blood they bow out and try to launder their conscience with a whistleblower-adjacent tweet.
Coxon’s observation that the frontier labs are speedrunning AI without the necessary guardrails in place is nothing new. “Move fast and break things” has been the unofficial mantra of Silicon Valley elites for decades.
But the real danger isn’t the technology. It’s the concentration of power and capital behind it. And we don’t need to look towards the future to find dystopia. It’s literally already here. In the US, Flock cameras surveil citizens, ICE is rounding up people and throwing them in cages (and sometimes killing or maiming them for life) with the help of companies like Palantir, which is also providing the tech for Israel to carry out what UN experts and humanitarian groups have called a genocide. Meanwhile Palantir co-founder and rumoured human, Peter Thiel, postulates that Greta Thunberg might be the antichrist.
And the timing of this latest departure could hardly have been better, providing a welcome distraction from a bombshell investigation published September 9 by The American Prospect that suggests Anthropic is working on a predictive surveillance tool to monitor activists and AI dissenters.
It seems Anthropic is hoist by its own petard. First they positioned themselves as the good-guy foil to OpenAI’s nefarious machinations. Then they were caught building a tool to surveil the very activists who would seek to hold them to account. Each new revelation highlights their hypocrisy.
While much of the world struggles to envision a future in which we can truly thrive, power is increasingly centralised in a few privately held companies with capabilities approaching those of states, without any of the attendant accountability. Now they are manoeuvring to control the parameters of the debate by silencing critics (see also Peter Thiel’s backing of ObjectionAI). And in a country where the average senator is pushing 65, regulatory capture looks increasingly likely, with lawmakers – who aren’t really cognisant of AI risks – more than willing to let industry decide how it should be regulated.
So whenever you see another former AI apparatchik rage-quitting in disgust, shouting down their ex-employer, it’s worth asking who benefits when the conversation shifts to a hypothetical robot apocalypse – conveniently distracting from the very real harms AI is already causing. Shelley wrote the ending two centuries ago, and the horror was never the monster. But good to know Anthropic is trying its best.
