Everyone’s talking about human extinction and AI regulation. But what’s behind it?
Laura Michelle Davis
AI doomsday scenarios have moved from fringe to mainstream after multiple researchers at Anthropic and Google resigned in rapid succession, warning that frontier AI labs are racing toward a greater risk of human extinction and gambling with our lives. Google DeepMind researcher Bilal Chughtai was the latest to resign on Monday, with the ominous caution that “AI has the potential to kill us all.”
The high-profile departure of Anthropic researcher Jacob Coxon last week was followed by public comments from OpenAI’s Sam Altman, who said the ChatGPT maker wouldn’t pursue an initial public offering in 2026 due to safety concerns.
In a widely publicized 3,800-word essay titled We Must Pace the Frontier, Anthropic CEO Dario Amodei urged tech companies to slow AI development, claiming that autonomous agents could take over the entire internet in less than a year unless clear guardrails are in place. Elon Musk, CEO of SpaceX and its subsidiary xAI, responded: “Dario is right.”
Meta CEO Mark Zuckerberg and Nvidia CEO Jensen Huang appear to be on the other side of the aisle, publicly rejecting calls for an industry-wide slowdown and arguing that market forces and legal liabilities are already sufficient to keep AI in check.
The bleak-toned debate over an AI-driven apocalypse has transformed into a tense political battleground over regulation and geopolitical competition. President Donald Trump dismissed extinction warnings as a hoax and part of a “sick conspiracy,” arguing that a “strong and smart” president is sufficient oversight, and that slowing domestic AI development would cede ground to China.
In Congress, lawmakers are drafting a series of bills — ranging from Bernie Sanders’ proposal to ban artificial superintelligence, to bipartisan “kill switch” and “no autonomous AI” measures — revealing disagreement over whether to halt, tightly control or merely monitor frontier systems.
Many critics say that focusing on the catastrophic dangers of AI models — which are portrayed as godlike, all-powerful machines — is part of a Big Tech marketing tactic aimed at boosting funding and attracting investors. And amid a growing backlash against AI, others highlight how talk of human extinction serves as a distraction from some of the real dangers AI poses in the here and now, from the climate and resource impacts of data centers to the use of AI-powered autonomous weapons in warfare.
The AGI bottleneck
AI industry leaders point to the immediate danger of “recursive self-improvement,” a hypothetical scenario in which artificial general intelligence, or AGI, enters a runaway feedback loop, optimizing its own code without human oversight. There’s never been a broad consensus on when (or if) LLMs will reach “superintelligence,” and many leading voices have called this forecast mostly hype.
Recent studies demonstrate that while AI agents can carry out narrow tasks like code execution, they consistently hit a wall when it comes to reasoning and judgment needed to achieve open-ended, empirical breakthroughs.
In a 13,000-word essay published in AI as Normal Technology, authors Sayash Kapoor and Arvind Narayanan analyze AI security risks, including recent so-called rogue AI incidents, such as the notorious Hugging Face hack by OpenAI agents. These kinds of “loss of control” occurrences — which bolster AI’s existential risk narrative — are largely viewed by cybersecurity experts as “consequences of companies failing to adopt basic security precautions” rather than “a sign of AI reaching a new milestone in cybersecurity.”
The far more immediate danger is an institutional feedback loop, where AI models are given real-world decision-making power, exercise increasing authority, and are relied upon more and more, according to Vishal Misra, vice dean of computing and AI at Columbia Engineering. “Slowing model development is one way to break that loop. It is not the only one,” Misra said.
In other words, the real risk isn’t just smart models running wild in a lab, but society recklessly delegating power to them.
The race with China
Proponents of fast-paced AI development argue that the US has to outpace China to maintain its military, intelligence and economic dominance. Presenting AI as a critical national security weapon helps create a sense of urgency that’s used to block new regulations and reject any slowdowns.
In his essay, Amodei framed the tech race as a battle between democratic AI and authoritarian AI, writing that “a Chinese lead in AI would pose grave danger for the United States and the world” by allowing China to “militarily dominate democracies.” Alex Karp, CEO of Palantir, which runs an AI-powered military “kill chain” called Maven, regularly plays the same tune.
In an interview with The View last week, Texas Sen. Ted Cruz called for more legislation to guard against the “catastrophic risks” of AI, but emphasized the risk of falling behind China. “If there are going to be killer robots, I would rather they be American killer robots, rather than Chinese killer robots,” he said.
If any of today’s bellicose rhetoric sounds familiar, that’s because it borrows heavily from the Cold War playbook. During the Cold War, the arms race hung on the notion that any technological gain by the Soviet Union was a direct threat to American survival. Today, AI is presented in the same zero-sum terms.
In 1983, President Ronald Reagan said in his Strategic Defense Initiative speech, “We maintain the peace through our strength. Weakness only invites aggression.” Just replace missile shields and nuclear weapons with compute power and frontier models — the similarities are clear.
The regulatory capture game
Some AI executives claim they want to regulate and slow down AI development to protect humanity, a concern that recently led Microsoft to create its Humanist AI code of conduct. However, critics argue this is a calculated strategy to block competition and dominate the market.
Regulatory capture would mean that AI companies actively shape the rules that are supposed to govern them — keeping Big Tech at a powerful advantage under its own guidelines before government institutions can enforce strict, meaningful laws.
It’s not unlikely that Anthropic is leaning into its safety-first brand as part of its IPO pitch, assuring investors that going public could improve safety standards. In Amodei’s essay, he argues that “Anthropic has long supported sensible and targeted AI regulation, specifically bills that focus on transparency and on third-party auditing.”
Mandating complex “third-party auditing” requires immense financial, legal and operational overhead. While Big AI labs can absorb these costs, startups, academic researchers and open-source developers cannot. In other words, it would be cost-prohibitive or legally impossible for smaller AI players to compete at the “frontier.”
And there’s an obvious escape clause: Amodei says that because traditional legislation moves too slowly to keep up with the rapid pace of AI progress, “AI companies can and should voluntarily work together to set standards” (emphasis added) — effectively a call for self-regulation.
“Any industry or corporation would prefer to self-regulate, i.e., not be constrained by regulation put in place to protect the rights and interests of the population at large,” said Emily Bender, co-author of the book The AI Con.
Focusing on the speculative pace of AI development distracts lawmakers from enforcing already-existing civil, privacy and environmental protections, Bender told CNET via email. At the same time, framing AI risks around sci-fi “rogue AI” scenarios creates a setup where tech insiders and engineers are treated as the sole authorities on safety, sidelining outside experts from shaping regulation.
That allows Big AI to avoid real accountability for their products’ environmental impact, economic disruption and dangerous behaviors.
More money, more power
Warnings that AI could destroy humanity might not be a deliberate publicity stunt, but there’s a convenient side effect: The technology sounds incredibly powerful. And that alters the status of AI labs, making them major global political players.
“They are no longer software companies selling products,” Misra told CNET. “They become custodians of a technology supposedly important enough that governments have to ask them how it should be regulated.”
Critics and financial journalists note that sounding the alarm about “existential risk” or “self-evolving superintelligence” keeps investors pouring money into Silicon Valley, while distracting from the industry’s diminishing returns and severe financial pressures.
Take, for one, the fact that AI models have become astronomical money pits. Tech titans pour hundreds of billions of dollars into AI capital expenditures (data centers, chips, power grids), yet direct revenue from these investments is a fraction of the costs.
Many frontier AI companies have been selling “tokens” (AI usage) to businesses at below-cost prices, effectively subsidized by venture capital. As enterprises actually scale their workflows, they find their AI bills skyrocketing, causing many to rethink their budgets.
Companies are starting to run out of high-quality, human-generated text on the internet to train their models. Out-of-the-box frontier models still struggle with actual reasoning. Current LLM technology is hitting a plateau. The AI bubble is revealing a gap between what investors hope AI can do and what it can actually monetize.
The financial math is becoming increasingly difficult to justify to Wall Street, so AI giants have to sell investors on the next big thing to keep valuations high. They can’t just be another company — they have to be building the future technological singularity capable of wiping out humanity.
______________________________________________________________________________
CNET’s Zachary McAuliffe contributed to this story.
The Latest from Laura
How AI Doomsday Talk Is Making Tech Giants More Powerful, Not Safer
Interest Rates Just Went Up. Here’s Why Consumer Tech Might Cost Even More
First 2 Deaths Reported in Cyclospora Outbreak. Get the Info You Need to Keep Yourself Safe
