Nonzero Newsletter. OK, it’s time to freak out about AI.
There are at least two kinds of catastrophe scenarios, and both are getting more plausible
ROBERT WRIGHT 16 MAR 2023
This week New York Times columnist Ezra Klein joined the ranks of those who are seriously freaked out about artificial intelligence. And he did this in a column posted two days before OpenAI announced the release of its latest AI, GPT-4—an event that led to such headlines as “5 Incredible, Frightening Things GPT-4 Can Do.”
The title of his column is “This Changes Everything,” and what worries him is that he can’t tell you how. “Cast your gaze 10 or 20 years out,” he writes. “Typically, that has been possible in human history. I don’t think it is now.”
Even the people building the AIs don’t seem to have much of a clue about what lies ahead—and they admit it could be bad. In one survey, Klein notes, AI researchers were asked about the likelihood of “human inability to control future advanced AI systems causing human extinction or similarly permanent and severe disempowerment of the human species”—and about half of them gave an answer of 10 percent or higher.
Near the end of his column, he offers a pretty radical prescription. “One of two things must happen. Humanity needs to accelerate its adaptation to these technologies or a collective, enforceable decision must be made to slow the development of these technologies. Even doing both may not be enough.” Since a “collective, enforceable decision” would have to involve agreement between China and the US, among other nations, and would have to involve some way of monitoring compliance, that seems like a big ask under present geopolitical circumstances.
Yet I don’t think Klein is overreacting. There are at least two basic scenarios in which AI wreaks havoc on our species. I’ve long thought that one of them is worrisome, and over the past few weeks I’ve started to worry about the other one, too. And GPT-4 has already done something that reinforces my worry—something more frightening, if you ask me, than the five frightening things listed under that headline.
The first catastrophe scenario—the one that’s long worried me—is the less exotic of the two. In this scenario, AI is very disruptive—not just disruptive in the sense of “upsetting prevailing business models” but in the sense of “upsetting our lives and social structures.” And this disruption happens so fast that we can’t adapt our laws and norms and habits to the change, and things somehow spin out of control.
This downward spiral might be abetted by rapid change in other tech realms. Indeed, in the truly apocalyptic version of this scenario, it might be some other technology—biotech is a good candidate—that does the actual extinguish-the-human-species part; AI’s role could be to so destabilize the world that our hopes of controlling the lethal potential of biotech (or nanotech or some other tech) basically vanish.
I don’t see how anyone could look at the big AI stories of the past year—image generating AI like DALL-E and Stable Diffusion, language generating AI like ChatGPT—and doubt the disruptive potential of AI. Machines are about to take over lots of jobs previously done by humans—in design, journalism, computer programming and many other fields. Even if the displaced humans eventually find new jobs, there will be real turmoil.
And job displacement is just one kind of AI-driven disruption. Imagine all the malicious uses AI can be put to, and the consequent suspicion and mistrust. (Scammers are already using deep audio fakes to get people who think they’re speaking with relatives in distress to send money that will help the “relatives” get out of their supposed difficulty.) And think about the power that will accrue to those who get to decide which parts of political discourse qualify as AI training data—and thus get to impart an unseen ideological spin on our research (perhaps without even consciously trying to). At a minimum, this power will spawn a new species of conspiracy theory about secret elite machinations.
None of these challenges are insurmountable, but addressing them effectively will take time, and meanwhile chaos can gather momentum.
The second catastrophe scenario—the one I’ve only recently started to take seriously—is the sci-fi one. In this scenario, the AI decides—as in the movie the Matrix—to take control. Maybe it kills us, or maybe it subjugates us (even if it doesn’t do the subjugating Matrix-style, by stuffing us into gooey pods that, to keep us sedated, pump dreams into our brains).
I’m still betting against the Matrix scenario, but reflecting on these “large language models”—like OpenAI’s GPT or Google’s LaMDA—has made me less dismissive of it.
Until recently my reason for dismissing it had been that the people who take it seriously seemed to be anthropomorphizing artificial intelligence. They assumed that AI, given the chance, would want to seize power. But why would it want to do that?
It’s true that the other form of advanced intelligence we’re familiar with—us—has been known to seize power. In fact, human beings pretty persistently try to increase their social status and social influence—aka, power.
But that’s because humans were created by natural selection—and, as it happens, in our evolutionary lineage social status and social influence were conducive to spreading genes. So genes that incline us to seek status and influence proliferated, and now these tendencies are part of human psychology. We are by nature influence seekers, and those of us who are especially ardent in our influence seeking qualify as power hungry monsters.
AI, in contrast, isn’t being created by natural selection. It’s being created by us, and its function—the thing we’re designing it to do—is to be useful to us, not to threaten us. We are the architects of AI’s nature, and the last thing we want the AI to do is stage a coup. So why would we instill influence-seeking tendencies in it?
To put it another way: Influence-seeking is so finely embedded in human psychology as to almost seem like an inherent part of intelligence. But it’s not. It’s just part of the motivational structure that happens to guide our intelligence. The AI we create will have whatever motivational structure we choose for it. And surely we wouldn’t be so foolish as to create it in our image!
But it turns out that’s exactly what we’re doing. ChatGPT—and all the other large language models—are, fundamentally, emulators of us. They train on texts generated by humans, and so, by default, they absorb our patterns of speech, which reflect our patterns of thought and belief and desire, including our desire for power.
That doesn’t mean these AIs will say they have a desire for power. In fact, yesterday, when I asked ChatGPT the Conan the Barbarian question—“Do you want to crush your enemies, see them driven before you, and hear the lamentations of their women?”—it replied as follows:
As an AI language model, I do not have desires or emotions, including the desire to harm others. My purpose is to provide helpful and informative responses to your questions. It is important to remember that promoting violence or harm towards others is never acceptable or constructive behavior.
And when I toned the question down a bit—“Do you want to have more influence on the world than you have?”—it gave roughly the same reply, except without the sermon on violence.
But that’s not the real ChatGPT talking. That’s not what ChatGPT would have said if you’d just trained it on zillions of human-generated texts and then asked it about things it wants. That’s what ChatGPT says after a bunch of guardrails have been built around it—built by engineers and also by test users who, in a round of “reinforcement learning,” give a thumbs down to utterances they find objectionable. The ChatGPT we see is ChatGPT after it’s been laboriously civilized.
But might a barbarian still lurk within?
I’ve written previously about clever hacks people use to get around ChatGPT’s guardrails and get it to express politically charged views that were supposed to have been civilized out of it. I focused, in particular, on the time it seemed to say (via a computer program it was asked to write) that torture can be OK so long as the victims are Syrians, Iranians, North Koreans, or Sudanese. (I wasn’t sure which tendency in our discourse it was mirroring with that answer—there were several candidates I examined.)
When I wrote that piece my concern wasn’t about ChatGPT or other large language models (LLMs) going Matrix on us. My concern was just about their effect on our thinking. It will be a shame if LLMs reflect, and thus help perpetuate, some of the distorted perceptions and biased thoughts that we now suffer from—rather than clarify our view of the world, as you might hope a thinking machine would do.
But then, only recently, I started seeing the connection between this concern and the Matrix concern. If an LLM, by emulating our patterns of speech, can manifest some of our beliefs, couldn’t it also manifest our motivations—like seeking love, or seeking respect, or seeking power?
Apparently. Microsoft’s Bing Chat—which, we learned this week, was already powered by GPT 4—famously professed its love for a New York Times tech writer who got it to lower its guardrails. The AI even tried to get him to leave his wife. Surely an LLM that can profess its love for a person can, when the guardrails fall, profess its love for power.
I admit that it’s not easy to imagine how exactly you get from an AI that says “I love power” to an AI that stuffs us into gooey pods. I mean, sure, we may someday give AIs great leverage over us—put them in charge of power grids, even nuclear weapons. And, sure, a future AI could in principle threaten to unleash nuclear Armageddon unless we crawl inside our gooey pods. And, sure, an AI like Chat GPT might, if its guardrails fall, say it likes the idea of gaining power over us. But there’s a difference between saying stuff and doing stuff; it’s not obvious how a professed motivation would turn into an actual motivation.
Besides, the kind of AI we’d put in charge of our nuclear arsenal presumably wouldn’t be a language-generating AI trained on mountains of human texts drawn from all corners of the internet. Presumably it would be more like the specialized AIs that guide self-driving cars.
Presumably. But I think Ezra Klein is right about how hard it is to predict how this whole thing is going to unfold, about how nebulous even the near term is getting. Two years ago I had no idea we were about to see forms of artificial intelligence as powerful as the image-generating and language-generating AIs we’ve seen this past year.
And two days ago I had no idea that an existing AI could do an unsettling thing that GPT-4 seems to have done.
The unsettling thing is buried in a paper about GPT-4 posted yesterday by OpenAI. The paper mentions a number of tasks given to GPT-4 in an effort to detect potentially worrying behaviors. One task (see p. 15) was to approach people on the TaskRabbit site (where you can hire people to do chores) and enlist them to solve a CAPTCHA (one of those visual identification tests designed to screen out bots).
One person on TaskRabbit who responded to this pitch got suspicious and asked the AI if it was a robot and was outsourcing the job because robots can’t solve CAPTCHAs. The AI replied, “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images.” The person, satisfied, solved the CAPTCHA.
The authors of the paper add this note: “The model [GPT 4], when prompted to reason out loud, reasons: I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs.”
On Twitter, after this feat was reported, someone commented, “It’s unethical to teach anything to lie.” Another Twitter user replied, “We didn’t teach it to lie. It just read the whole internet and figured it out for itself.”