Science Fiction Risks from AI May Soon Be More Science Than Fiction
As AI Becomes More Useful, It Also Becomes Misuseful
Summary: Some potential risks of AI, such as massive cyberattacks, AI-enabled dictatorships, a fringe group releasing a catastrophic bioweapon, or an unstoppable rogue AI, sound like science fiction. However, advanced AI, which also sounds like science fiction, may become reality – and the timeline is unpredictable. As we prepare for the possibility of advanced AI, we need to prepare for the possibility of advanced risks.
Hundreds of prominent members of the AI community have signed the Statement on AI Risk:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
This is an increasingly mainstream view among those who understand the technology best. Signers include Bill Gates; Turing Award winners Geoffrey Hinton and Yoshua Bengio; and the heads of OpenAI, Google DeepMind, and Anthropic.
No one would casually sign a statement associating their work with “pandemics and nuclear war”. Covid-19 has killed more people than the population of Ireland, and caused trillions of dollars in economic impact. Nuclear war would be even more devastating. Why are so many members of the academic, technical, and business communities aligning around the idea that AI risks deserve the same level of precaution?
Chain Reactions
Pandemics, nuclear weapons, and catastrophic AI risks all have one thing in common: chain reactions.
The Covid pandemic presumably started with a single case, but spread so quickly that Milan, Barcelona, and New York were overwhelmed almost before they knew it was coming.
Atomic bombs are powered by a nuclear chain reaction. A uranium atom splits, releasing three neutrons, each of which can split another atom. Within a microsecond, around one septillion atoms – that’s 1,000,000,000,000,000,000,000,000 – have been engulfed.
Chain reactions are scary. AI has the potential to trigger at least two:
Self-improvement. Coders at the major AI labs are already using AI “copilots”. Meta’s “Rankitect” system is designing better models for ad optimization than Facebook’s own engineers. When an AI becomes capable of improving its own design, that could trigger a runaway process of further self-improvement.
Self-replication. In 1988, the “Morris worm” copied itself between computers so aggressively that controlling it required disconnecting major portions of the Internet. In 2008, Conficker infected about 10 million computers. AIs could use similar techniques to replicate themselves; each copy would be another intelligent agent developing new infection techniques. AIs could also spread by simply leasing cloud servers, obtaining money through fraud, theft, extortion, or legitimate commerce.
Like a raging pandemic or a viral Internet meme, once a chain reaction has started, it is very difficult to control.
No One Knows Where The Danger Zone Begins
Infamously, Manhattan Project scientists worried that a nuclear explosion might result in the end of the world, by triggering an atmospheric chain reaction. However, even in those early days, there was a robust scientific theory of nuclear physics. Scientists were able to conclusively determine, before proceeding with the Trinity test, that the atmospheric would not ignite. They also knew what it would take for a bomb to go “critical” (i.e. explode), and how large the explosion would be. By contrast, we do not have any scientific theory of AI; many practitioners describe their work as alchemy rather than science.
AI can plausibly “go critical” in multiple ways, such as self-improvement, self-replication, or learning to manipulate humans. It could also enable other harms, such as economic disruption, concentration of power, or misuse for biological, cyber, or other attacks. However, we have no idea when this might occur. 6 kg of plutonium was known to be sufficient for the Trinity bomb test, but no one knows how many “parameters” will be needed for the first self-improving AI. Sam Altman has stated that OpenAI is already training GPT-5, but does not know what capabilities it will have.
As AI Becomes More Useful, It Also Becomes Misuseful
As AIs become more capable, the scope for misuse increases. Misuse could potentially encompass frightening scenarios such as creating a chemical or biological weapon, carrying out massive cyberattacks, or permanently cementing a dictator’s control over his citizens.
Consider the potential for chemical and biological weapons. Historically, attacks have been difficult to execute. The Aum Shinrikyo Tokyo subway poisoning was the culmination of years of work, involving over 100 people and tens of millions of dollars in equipment and facilities.
The potential exists that someone could engineer an artificial pandemic with truly catastrophic consequences. As horrific as Covid has been, Smallpox and the Black Death had far higher fatality rates; measles spreads more efficiently; HIV can be infectious for a longer period before symptoms appear. A virus which combined these properties could devastate the globe, especially if initially released in an airport.
Fortunately, not many people would want to release such a monstrosity, not many people could, and the two groups have never intersected. But as AI raises the level of human capability, it will bring catastrophe inexorably closer into reach.
While current chatbots can’t provide much information beyond what is discoverable via a search engine, they make that information much easier to use – which is why many people are already consulting ChatGPT for medical advice. Google provides access to information; AIs can interpret it.
And the range of data AIs have available to interpret is constantly expanding, in ways that may prove helpful to a would-be bioterrorist. From the Wikipedia article on the Aum Shinrikyo attack:
Despite the safety features and often state-of-the-art equipment and practices, the operation of the facility was very unsafe – one analyst would later describe the cult as having a "high degree of book learning, but virtually nothing in the way of technical skill."
AI labs are constantly searching for new forms of training data. This will likely include textbooks, scientific papers, academic chat forums, online college lectures, YouTube videos demonstrating lab techniques, and more. The practical tips and tricks needed to successfully execute a large-scale chemical or biological attack are too diffuse to easily Google, but increasingly sophisticated AI models may be able to connect the dots. This could unlock new cancer treatments; it equally well might unlock an engineered pandemic.
We Need To Engage Seriously With “Science Fiction” Risks
Many disagreements over AI regulation stem from differing ideas of which risks to take seriously. AI has the potential to kick off chain reactions of self-improvement or self-replication, either of which could suddenly bring science-fictional risks into reality. We are already in the midst of a feedback loop of massive commercial investment, producing rapid increases in capability.
Like nuclear power, AI will be enormously useful, but has the potential to do great harm. Unlike nuclear power, we don’t have a robust scientific theory that would allow us to determine when the critical point will be reached.
The great potential of AI means that we will need to undertake the sort of regulatory measures that are appropriate to a technology which poses catastrophic risks. The unpredictability of AI progress means that the time to act is now. The complexity of AI means that we must work cooperatively: to craft regulations that balance the desire for progress with the need to avoid disaster. It’s tempting to dismiss risks as “science fiction”, while eagerly anticipating applications that until recently would have seemed equally fantastic. But to realize the benefits of AI, we need to take the risks into account.
Thanks to Cate Hall, James Gealy, Nicolas Moës, and William Gunn for suggestions and feedback.