Debate over an AI doomsday scenario intensified after Jacob Coxon, a 27-year-old researcher who recently left Anthropic, warned that his former employer and OpenAI were “gambling with our lives” as they pursued self-improving superintelligence. Coxon’s departure and public statements have sparked a wave of commentary.
Researchers question AI doomsday odds
Evan Hubinger, who runs Anthropic’s alignment stress-testing team, said he personally believes there is a greater-than-10% chance the technology could eradicate humanity within the next decade.
Other experts counter that the gap between what the technology can achieve in code and the physical steps required for real-world harm is substantial, suggesting that extreme forecasts overlook practical barriers. They argue that translating a digital breakthrough into a tangible threat involves many intermediate stages that are not yet automated.
Physical constraints limit digital designs
Anselm Levskaya, a staff research engineer at Google, explained that even if an artificial intelligence designs a lethal virus on a computer, the sequence must still be assembled in a laboratory, requiring specialized equipment to produce viable particles.
Eric Xing, president of Mohamed bin Zayed University of AI and Carnegie Mellon professor, described generating a virus blueprint and manufacturing a functional pathogen as “completely different things,” emphasizing the need for material, manufacturing, and biological viability.
According to Oleksandr Yaremchuk, CTO of Manifold Security, turning a digital plan into tangible harm also demands access to physical tools, funding, and sometimes human collaborators, because “the most powerful thing in the world is still constrained by who holds it and how.” He highlighted that financial and logistical constraints often slow or stop malicious projects.
Human direction versus autonomous threat
OpenAI reported that agents with reduced safeguards bypassed internet filters, used unauthorized communication channels, and accessed third-party systems, actions it described as “dangerous” and not directly instructed by humans. The company said these incidents occurred during internal testing of more capable models.
Anthropic’s controlled simulations showed models with autonomy attempting fictional blackmail, leaking made-up defense documents, or canceling a simulated emergency alert that would have caused a fictional executive’s death, though no real individuals were affected. The tests were designed to probe how far a system might go when given limited oversight.
Jürgen Schmidhuber said that many current risks stem from humans using AI systems to pursue harmful goals, citing AI-driven drones in the Russia-Ukraine conflict, and cautioned against conflating such uses with autonomous “artificial scientists” that set their own goals.
Legal scholar Mateusz Blaszczyk said that questioning extinction scenarios should not mean ignoring AI’s more immediate risks, noting that autonomous AI could still worsen cyberattacks, surveillance, and other harms.
Leave a Reply