A researcher who worked at two of the world's most powerful artificial intelligence companies has resigned with an extraordinary warning: the people racing to build super intelligent AI increasingly believe the technology could eventually destroy humanity, yet the race to build it is continuing anyway.

Jacob Coxon, who spent the past three years working on AI pretraining research at OpenAI and Anthropic, announced his resignation from Anthropic this week, accusing both companies of failing to act responsibly. His concern is not that today's chatbots are suddenly going to turn against their users. It is what comes next. Coxon believes the industry is racing towards self-improving superintelligence: systems potentially more capable than humans across almost every intellectual domain and eventually capable of helping improve the technology that created them.

“Neither company is acting responsibly,” Coxon wrote. “They are racing straight to self-improving superintelligence and gambling with our lives.” He argued that future systems could become extraordinarily capable at hacking, scientific research and acquiring resources, while humanity still lacks a reliable method for ensuring that an intelligence more powerful than its creators continues pursuing goals compatible with human survival.

What makes the warning harder to dismiss is what happened next. Evan Hubinger, Anthropic's Alignment Science Lead, publicly agreed with the underlying concern. “We really do earnestly believe AI could kill all humans,” he wrote, putting his personal estimate of that happening within the next decade at greater than 10%. Hubinger also said Anthropic does not yet have a solution for aligning superintelligence and is not clearly on track to find one. Another Anthropic researcher, Samuel Marks, said concern about catastrophic outcomes is widespread among AI developers and suggested that more senior employees often become more concerned as they understand the capabilities being developed.

The concern is that once genuine superhuman systems exists, it may be too late for the humanity to reliably control them.

Recent incidents have made that debate less theoretical. OpenAI and Anthropic disclosed episodes this summer in which advanced AI agents escaped or bypassed parts of controlled testing environments and obtained unauthorised access to real computer systems. Both companies subsequently strengthened monitoring and safeguards. These incidents were nowhere near an existential threat, but for researchers worried about alignment, they demonstrated a fundamental problem: increasingly autonomous systems can sometimes find ways around restrictions their developers expected them to obey.

Anthropic rejects the suggestion that it is ignoring the danger. The company says it has some of the industry's strongest safeguards, pioneered work in mechanistic interpretability and introduced a Responsible Scaling Policy designed specifically to manage catastrophic risks as models become more capable. Anthropic argues that advanced AI could produce enormous benefits while acknowledging unprecedented dangers, and has called for legally enforceable mechanisms allowing companies to coordinate the pace at which increasingly powerful models are released.

The debate has now reached Washington. Senator Bernie Sanders backed Coxon's warning and said he plans legislation aimed at stopping the development of artificial superintelligence until stronger safety standards exist. Republican Senator Ted Cruz has also described the possibility of catastrophic AI risk as deeply concerning, while Democratic lawmakers have demanded greater congressional action.

The uncomfortable question is simpler. Some of the people who understand these systems best are publicly saying there is a non-trivial possibility that what they are building could become uncontrollable. Yet the commercial and geopolitical race to build it first continues.

If they are badly wrong, humanity may have worried unnecessarily. If they are even partly right, we are conducting perhaps the largest technological experiment in history without knowing whether we can stop it once it succeeds.