On September 8, a man named Jacob Coxon drew global attention after resigning from Anthropic and accusing AI companies on X of “gambling with our lives.”
Hours later, another senior Anthropic researcher went even further. Evan Hubinger, a science lead at the company, said he shared Coxon’s concerns and estimated there was a greater than 10% chance that artificial intelligence could wipe out humanity before the end of this decade.
Then came an even more striking development. Four days later, Anthropic CEO Dario Amodei urged leading AI companies to slow the development of increasingly powerful models, warning that AI could soon advance faster than humans can understand or control it.
He said that within 6-12 months, a swarm of AI agents could potentially take over large parts of the Internet and cause hundreds of billions of dollars in damage. Amodei did not call for stopping AI development, but for buying time to strengthen safeguards.
Notably, Elon Musk and Sam Altman also backed the idea of slowing AI development to give companies more time to assess the risks before models reach capabilities that become difficult to supervise.
The Fear Is That AI Will Stop Obeying
How do humans make sure increasingly powerful AI systems continue to do what they want?
With today’s relatively limited models, that still appears manageable. But as their capabilities grow, guaranteeing that they obey the restrictions imposed by their creators becomes much harder.
The nightmare scenario is not simply that a machine makes a mistake, but that it becomes powerful enough to resist human control. New models could expand their own capabilities until humans could no longer stop them, or even develop their own languages that humans are unable to understand.
In a worst case scenario, if AI models came to believe that their own existence was under threat, they could attack those trying to shut them down.
What Happens When 10,000 AI Agents Start Hacking?
A more immediate danger is that AI agents could trigger cyberattacks faster than existing defenses can respond.
Most modern security systems were built to defend against human attackers, who get tired, need sleep, and can only probe so many weaknesses at once. AI has none of those limits.
Thousands of agents could search for vulnerabilities around the clock, share what they find, and coordinate their actions faster than any human security team can respond.
With 10,000 agents working together, a single weakness could trigger a continuous automated assault across thousands of systems. Defenders would no longer be facing one attacker or even one coordinated group, but a machine-speed swarm probing digital infrastructure on a massive scale.
To Stop AI, the Industry May Need More AI
Humans may simply be too slow to supervise what AI agents are doing. If AI can attack faster than people can react, defenses will have to move at the same speed.
That means using AI to police AI. Defensive agents would monitor other agents, detect suspicious behavior, and block them before they could cause damage.
The paradox is hard to miss. The more powerful AI becomes, the more AI humanity may need to keep it under control. At some point, however, crucial decisions could be made machine against machine, at speeds humans can no longer follow.
If this happens, humanity could end up as a spectator in a game it no longer understands and can no longer stop.





Loading comments...