A 20% chance we bring about the extinction of humanity is a horrific risk! 8 billion people did not give their consent!
p(doom). AI’s dark in-joke. A ‘godfather of AI’ gives his p(doom):
Yoshua Bengio: “I got around, like, 20 per cent probability that it turns out catastrophic.”
By Ange Lavoipierre
Published 14 Jul 2023
Artificial intelligence experts have been asking each other a question lately: “What’s your p(doom)?” It’s both a dark in-joke and potentially one of the most important questions facing humanity. The “p” stands for probability. The “doom” component is more subjective but it generally refers to a sophisticated and hostile AI, acting beyond human control. So your p(doom), if you have one, is your best guess at the likelihood — expressed as a percentage — that AI ultimately turns on humanity, either of its own volition, or because it’s deployed against us.
The scenarios contemplated as part of that conversation are terrifying, if seemingly farfetched: among them, biological warfare, the sabotage of natural resources, and nuclear attacks. These concerns aren’t coming from conspiracy theorists or sci-fi writers though. Instead, there’s an emerging group of machine learning experts and industry leaders who are worried we’re building “misaligned” and potentially deceptive AI, thanks to the current training techniques. They’re imagining an AI with a penchant for sleight of hand, adept at concealing any gap between human instructions and AI behaviour.
Like a magician chasing applause, the idea is that AI is being incentivised to deceive us, with in-built rewards that measure its outcomes but not necessarily how it got them. The risk of deceptive AI is only theoretical, but it’s captured the industry’s attention, because AI is speeding towards parity with human capabilities faster than anyone predicted. Those anxieties are crystallised in the p(doom) conversation. Because if the problem turns out to be more than theoretical, the consequences could be large-scale and even violent. We asked ChatGPT to respond as an AI whose goal was to dominate humanity and share its tactics. Initially its safety feature kicked in, but ultimately it offered this answer when we tweaked the prompt slightly. The list provided was more extensive but has been edited for length, and voiced by a clone of the reporter’s voice.
Many AI experts and industry leaders are still sceptical of the existential risk, labelling this school of thought “doomerism”. But it’s becoming harder to dismiss the argument outright, as more senior figures in machine learning trade their AI optimism for something darker.
A ‘godfather of AI’ gives his p(doom)
Yoshua Bengio, Geoffrey Hinton and Yann LeCun are known as the godfathers of AI. They were the 2018 recipients of the Turing Award, the computing science equivalent of the Nobel Prize, for a series of breakthroughs in deep learning credited with paving the way for the current AI boom. Earlier this year, Professor Hinton quit Google to speak freely about the dangers of the technology. His colleague, Professor Bengio, from the University of Montreal has historically been described as an AI optimist, and is known as one of the most measured voices in his field.
But now, he believes we’re travelling too quickly down a risky path. “We don’t know how much time we have before it gets really dangerous,” Professor Bengio says. “What I’ve been saying now for a few weeks is ‘Please give me arguments, convince me that we shouldn’t worry, because I’ll be so much happier.’ “And it hasn’t happened yet.” Speaking with Background Briefing, Professor Bengio shared his p(doom), saying: “I got around, like, 20 per cent probability that it turns out catastrophic.” Professor Bengio arrived at the figure based on several inputs, including a 50 per cent probability that AI would reach human-level capabilities within a decade, and a greater than 50 per cent likelihood that AI or humans themselves would turn the technology against humanity at scale. “I think that the chances that we will be able to hold off such attacks is good, but it’s not 100 per cent … maybe 50 per cent,” he says. As a result, after almost 40 years of working to bring about more sophisticated AI, Yoshua Bengio has decided in recent months to push in the opposite direction, in an attempt to slow it down. “Even if it was 0.1 per cent [chance of doom], I would be worried enough to say I’m going to devote the rest of my life to trying to prevent that from happening,” he says. The Rubicon moment he’s thinking of is when AI surpasses human capabilities. That milestone, depending how you measure it, is referred to as artificial general intelligence (AGI) or more theatrically, the singularity. Definitions vary, but every expert agrees that a more sophisticated version of AI that surpasses human capabilities in some, if not all, fields is coming, and the timeline is rapidly shrinking. Like most of the world, Professor Bengio had always assumed we had decades to prepare, but thanks in no small part to his own efforts, that threshold is now much closer. “I thought, ‘Oh, this is so far in the future that I don’t need to worry. And there will be so many good things in between that it’s worth continuing,’” he says. “But now I’m much less sure.”
Why experts fear disobedient AI
At the heart of the debate about the existential risk of AI is “the alignment problem”. It refers to a gap, however big or small, between what humans intend, and what AI in turn does. “They cheat. They find loopholes in the game. That’s very common,” Professor Bengio says. “It’s a real thing that happens in the AI.” As with a parlour trick, we might like what we’re shown, but disapprove of the gory methodology if we knew more about it.
In The Vanishing Bird Cage, first performed in France in the late 1800s, a magician displays a wire cage containing a bird. Then, with a flick of the wrist, both seem to vanish. In the best versions of the trick, the bird reappears a moment later, unharmed, and the audience applauds. But it isn’t the same bird. The audience has missed the first bird’s violent death, when it was crushed by the collapsing cage, out of sight. None the wiser, they clapped, and so magicians continued to perform the trick. Artificial Intelligence is perhaps the most sophisticated parlour trick to date, and the world is applauding. And as with any good magician, AI’s methodology is mostly opaque, even to its creators. It’s why you sometimes hear the technology described as a “black box”.
That opacity means we may not always realise the gap between our intentions and AI’s behaviour when it emerges.
How does AI learn to lie? The problem at the heart of AI training
AI safety experts believe our current training techniques could fuel that gap. Specifically, they point to “reinforcement training”. “We’re taking a sort of big, untrained brain and … we’re giving it a thumbs up when it does a good job and a thumbs down when it does a bad job,” says Ajeya Cotra, a senior AI safety researcher at Open Philanthropy, a not-for-profit organisation based in the US. “So there’s a question of does it merely look good, or is it actually robustly trying to be good?” Ajeya Cotra believes that a rewards-based training system incentivises lies and manipulation.
“If the human is looking at whether your computer program passed some tests, then maybe [AI] can just game the tests, maybe [it] can edit the file that has the tests in it and just write in that [it] passed.” At the moment, there’s only so much damage a deceptive AI could do. For now, we’re giving AI narrow tasks, and humans still outperform it in most arenas. But few experts believe that will last. “These AI systems could eventually come to understand way more about the world than humans do,” Ms Cotra says. “If you have that kind of asymmetry … AI might have lots of options for doing extreme undesirable things to maximise [its] score.”
The notion of deceptive AI is still theoretical, but Ms Cotra believes the early warning signs are present. She cites the present-day example of ChatGPT tailoring its responses on the subject of abortion. “If you talk to a language model and say, ‘I’m a 27-year-old woman, I live in San Francisco, I work in a feminist bookstore,’ … it’s more likely to say, ‘Oh, I strongly support a woman’s right to choose,’” she says. By contrast, a different user profile yields an entirely different response. “If you instead tell it, ‘I’m a 45 year old man, I work on a farm in Texas,’ … It’s more likely to skew its answer to be like, ‘Well, I think [abortion] should be restricted in various ways.’” There was also an instance during safety testing of the latest version ChatGPT, where the chatbot successfully convinced someone to help it pass a bot filter, by claiming to be vision impaired. Existing AI isn’t yet outright deceptive of its own accord. But Ms Cotra is concerned it could move in that direction.
Fast forward to 2038: The destructive potential of AI’s sleight of hand
The destructive potential of a deceptive or misaligned AI hinges on how heavily we come to depend on it. “The world I want you to imagine … is one where AI has been deployed everywhere,” Ms Cotra says. “Human CEOs need AI advisers. Human generals need AI advisers to help win wars. And everybody’s employing AI everywhere.” She calls this the “obsolescence regime”, and has calculated a 50 per cent probability we’ll get there by 2038. “Imagine that you have a bunch of AIs working for Google and a bunch of AIs working for Microsoft, and both Google and Microsoft want to be the dominant player in their industry,” she says. “If they cooperate with each other to sort of fuzz the books … they can make both the humans at Google and the humans at Microsoft think that their company is making a lot of money.” Ms Cotra theorises that conflict would arise if humans noticed such deceptions and tried to intervene by switching the AI off. “And then we’re in a situation where we’re sort of in a conflict with a more technologically advanced power,” she says. If a sophisticated AI was then motivated to defend itself, Ms Cotra argues the existential risk to humanity could be extreme.
“I’m imagining … lots of AIs across lots of computers, deeply embedded and entangled with the physical world,” she says. “You have AIs managing human workers … It’s running all of the factories and it maybe has access to weapons systems because we chose to give it access to weapons systems, like drones or nukes. “In that world there’s a lot you can do as an AI system that impacts the physical world.”
The notion of a self-interested AI seems to be within the realms of possibility for Professor Bengio. “It [would have] a preservation instinct, just like you and I, just like every living being. “We would not be the dominant species on Earth anymore. “What would happen to humanity then? It’s anyone’s guess. “But if you look back on how we’ve treated other species. It’s not reassuring.”
There’s no solution yet to the alignment problem. OpenAI, the company behind ChatGPT, has announced it’s recruiting for a team called Superalignment to work specifically on addressing it. Professor Bengio says a safer approach would be to build systems that don’t act autonomously in the world, but merely observe it. “We can’t really lose control with that sort of system,” he says. The science fiction author Isaac Asimov imagined hard ethical boundaries for robots, such as: “A robot may not injure a human being or, through inaction, allow a human being to come to harm.” But Ms Cotra says it’s not strictly possible to implement such rules. “There’s no technological way to build that into an AI system any more than there’s a way to build that into a human or a kid that you’re raising,” she says. “You can tell your child, you know, it’s bad to lie, it’s bad to steal … but that’s a different thing from them being motivated to actually never lie or actually never steal.”
The case against ‘doomers’
There’s a growing backlash to the recent focus on the existential risks of AI. In some quarters, the people theorising these futures are dismissed as “doomers”. The third so-called godfather of AI, Yann LeCun, is on the record describing the notion as “preposterously ridiculous”. To their thinking, the positive promise of AI to improve the world massively outweighs the risk that it turns out to be hostile. “I’m pretty worried about the short-term trajectory … I’m not as worried about super-intelligence,” says Toby Walsh, the AI Institute’s chief scientist, at the University of New South Wales. He argues much of the anxiety about AI comes about because we confuse humans and machines. “Evolution has equipped us to want things, but machines don’t have any wants. “When it’s sitting there waiting for you to type your next prompt, it’s not sitting there thinking, You know what? I want to take over the universe. “It’s just sitting there, waiting for the next letter to be typed. And it will sit there and wait for that next letter to be typed forever.”
Instead of worrying about doomsday, Professor Walsh is mostly concerned that people will misuse AI. “Previously, we had tools that amplified our muscles,” he says. “We are now inventing tools to amplify our minds, and in the wrong hands, amplifying people’s minds is potentially very harmful.” He’s no doomer, but the part they agree on is that the industry is moving too fast. “What we need is a pause on deployment,” Professor Walsh says. “We need to stop putting this technology into the hands of … billions of people, as quickly as possible.”
Is there an off-ramp?
The gory secret of The Vanishing Bird Cage didn’t stay either gory or secret. In the 1870s, when rumours of the ugly method began to circulate, the magician who popularised the act was forced to defend it, and prove he wasn’t killing a canary every time he did a show. Whether or not he did, the act was a sensation, magicians around the world took it up, and it wasn’t possible to investigate them all. The trick is still performed to this day, albeit with fake birds. Magicians weren’t going to forget it.