The story of how Geoffrey Hinton became “the godfather of AI” has reached mythic status in the tech world. While he was at the University of Toronto, Hinton pioneered the neural network research that would become the backbone of modern AI. (One of his students, Ilya Sutskever, went on to be one of OpenAI’s most influential scientific minds.) In 2013, Hinton left the academy and went to work for Google, eventually winning both a Turing Award and a Nobel Prize. I think it’s fair to say that artificial intelligence as we know it, may not exist without Geoffrey Hinton. But Hinton may be even more famous for what he did next. In 2023, he left Google and began a campaign to convince governments, corporations and citizens that his life’s work – this thing he helped build – might lead to our collective extinction. And that moment may be closer than we think, because Hinton believes AI may already be conscious. But even though his warnings are getting more dire by the day, the AI industry is only getting bigger, and most governments, including Canada’s, seem reluctant to get in the way. So I wanted to ask Hinton: If we keep going down this path, what will become of us? Mentioned: If Anyone Builds It, Everyone Dies: The Case Against Superintelligent AI, by Eliezer Yudkowsky and Nate Soares Agentic Misalignment: How LLMs could be insider threats, by Anthropic Machines Like Us is produced by Mitchell Stuart. Our theme song is by Chris Kelly. Video editing by Emily Graves. Our executive producer is James Milward. Special thanks to Angela Pacienza and the team at The Globe and Mail. Support for Machines Like Us is provided by CIFAR and the Max Bell School of Public Policy at McGill University.
TRANSCRIPT
Hi, I’m Taylor Owen from the Glob and Mail. This is Machines Like Us. [Music] If you listen to this show, I assume you pay at least a little attention to the world of AI. And if you follow AI, then you’ve almost certainly heard of Jeffrey Hinton. At this point, Hinton’s story has become almost mythlike. While he was at the University of Toronto, he developed the foundations of modern artificial intelligence. And then after a career in academia, he went to work for Google in 2013, eventually winning both a touring award and a Nobel Prize. I think it’s fair to say that artificial intelligence as we know it may not exist without Jeffrey Hinton. But Hinton may be even more famous for what he did next. In 2013, he left Google and began a campaign to convince governments, corporations, and citizens that the thing he helped to build might lead to our collective extinction. He’s been sounding that alarm for more than 2 years, and now he thinks AI may already be conscious. But Hinton isn’t just worried about our potential annihilation. He also believes that we’re on the brink of mass unemployment, that our banking system is in jeopardy, and the machines are already poisoning our information ecosystem. But even though his warnings are getting more dire by the day, the AI industry is only getting bigger and most governments, including Canada’s seem reluctant to get in the way. So I wanted to ask Hinton, if we keep going down this path, what will become of us? [Music] Jeffrey Hen, thanks for being here. Thanks for inviting me. So, I have to say I’m struck with a bit of cognitive dissonance talking to you like you’re you’re talking about the end of humanity and we’re just going to have a sort of casual conversation here about a a technology. I mean, it feel I feel the dissonance. Do you? It is a big cognitive dissonance. Um, I don’t know if you’ve seen the movie Don’t Look Up, but it’s it’s quite similar to what happens in that movie. Yeah. And like, how do you navigate that? How do you see your position in this? I just do the best I can. Um, I’m not a doomer in the sense that I think it’s more or less inevitable we’ll be wiped out. Like Yakowski, who just published a book saying with a title something like, “If anyone builds it, we all die.” I think that’s crazy. Yeah, it was it was fairly categorical. I think that’s crazy. We just don’t know. We’re coming to a time when we’re going to have things more intelligent than us. Most of the experts agree that that will happen in between 5 and 20 years and we just don’t know what’s going to happen. So the main the main point is we don’t know and we should be doing as much as we can to make sure good things happen. And I want to talk about that what we don’t know, why we don’t know it, what things we could do to to to stop it. The the one thing that seems clear though is that over the time you’ve been making these arguments and sounding these alarms, the pace of evolution of the technology has just exploded. It has only increased. How do you cope with that disconnect? I mean, just between this the level of your warning and the severity of it and the fact that really very little friction is happening. In fact, quite the opposite. We’re this industry is exploding. So you have to realize there’s a big difference between something like AI and something like nuclear weapons. Nuclear weapons only good for blowing things up. AI has many many very good positive uses in healthcare and education in almost any industry where you have data and you’d like to predict things. We’re not going to stop it because of all those good applications. So we’re in a very tricky situation where some people are calling for us to slow down or stop it. And I don’t think that’s particularly realistic. I think what we have to do is try and figure out how we can live with it. And it’s very hard to really get the full impact of it emotionally. I talk about it, but there’s only one area I think in which I’ve really absorbed it emotionally, and that’s in the threat to banks of cyber attacks. So I and I spread my money between three different Canadian banks because I’m convinced that there’s a good chance I mean it’s not necessarily going to happen but there’s a good chance that we’ll get cyber attacks designed by AI where we’ve got no clue what’s going on and they could bring down a Canadian bank. I assume that if they do the other banks will get a lot more cautious. And is this because of a ch uh a new technological capacity that you’ve seen emerge or is it because you’ve seen an increase in cyber threats based on that technology or what’s what’s signaling that change in behavior? So there’s been a huge increase in things like ransomware attacks. Also between 2023 and 2024 there was like a 1200% increase in fishing attacks. Um, I’m now getting a lot of spear fishing attacks where they use details about me to make the it look convincing. Um, I often have to ask an IT guy, is this male real? And he always says, no, it’s a spear fishing. Um, but what worries me even more is a very good researcher called Jacob Steinhard predicted in about 5 years time AI may well be designing its own cyber attacks that we have no understanding of. I want to come back to some of the risks you’ve identified and what some of their technological origins or causes are. Um, but this has been a process for you and coming to both understand these harms and then to decide to speak out about them. And I wonder if you could walk us through a little bit of that process is at what point did you really become concerned that the technology you were working on that you played a role in founding had these potential downside risks? I only got really concerned at the beginning of 2023. Um, I should have been concerned earlier, but I was having too much fun doing the research and I thought it would be a long long time before we had things smart as us that had sort of general intelligence like us. The advent of the large language models, GPT3, um, models at Google, um, like Palm that could explain why a joke was funny. So that was one ingredient and the other ingredient was research I was doing at Google on whether you could make analog computers um do large language models um which would require a lot less electricity and I realized there’s a huge superiority that digital models have which is that different copies of the same digital model can share what they’ve learned from their different experiences very efficiently and people can’t. If I experience something and I want to tell you what I’ve learned, I produce some sentences. That’s a very slow way to communicate information. If two digital intelligences um that are the same neural network experience different things and if they’ve got a trillion connections, they’re sharing information at like a trillion bits per sharing. Whereas when I give you a sentence, there’s maybe a 100 bits. So even if you perfectly understood what I said, you’re getting a 100 bits, whereas these digital intelligences are getting like a trillion bits. That’s a difference of more than a billion, a factor of a billion. So they’re incredibly much better at sharing information than we are. What do you mean by a they? What’s the they there? large lang large large language models a neural net running on some hardware they can share information much much more efficiently which is why things like GPG4 or Germany 2.5 or um claude know thousands of times more than a person cuz many different copies are running on different pieces of hardware suppose we had 10,000 people and they all went off to university and each person did one course but as they were doing the courses they could share what they’d learned so at the end even though each person’s only done one course all of them know what’s in all 10,000 courses that would be great and that’s what digital intelligences have and we don’t and why did you only see that particular capacity in 2023 I was just slow at seeing things it was cuz I was focusing on analog computation and my attention was drawn to the ways in which analog computation is not as good as digital computation so our brains are basically analog And the problem with analog computation is that to make it efficient, you need to use all the quirks and idiosyncrasies of the connectivity and the particular neurons in your brain. So the connection strengths in my brain are designed to work well with my neurons. Now you have different neurons and you have different wiring in detail. And so the connection strengths in my brain are of absolutely no use to you. Whereas for different digital copies of the same neural net, they work in exactly the same way. That’s what the digital is all about. And so they can share what they learn very efficiently. What was that realization like when you did see the potential of that? It was fairly shocking realizing that yes, that’s why they can learn so much more than us. In addition to that, I suspect now that the learning algorithm that they use called back propagation um may be better than the learning algorithm the brain uses. For a long time, people thought you’ll never be able to recognize real objects in real images using this dumb back propagation technique. And for a long time, we couldn’t because the nets were too small. But then we got much faster computers and we got much more data and then it worked amazingly well. And so it was it was it was the argument that if you build this bigger and bigger and bigger with more and more computation there will be some breakthrough that make that changes this thing in kind not in a evolutionary way but in a revolutionary way in a sense. Yes. And that’s what’s happened with these large language models. As you make them bigger and bigger you suddenly start getting new kinds of ability that you didn’t have before. When you describe that technical process it feels like something we should be able to understand. Why is it that we don’t know what these models are doing when they’re behaving? Okay, so it’s not like normal computer software. In normal computer software, you put in lines of code and you know what those lines of code are meant to do. And so you sort of know how it’s meant to behave. Here, you just put in some lines of code that tell it how to learn from data. What it learns depends on what structure there is in the data. And there may be all kinds of structure that you weren’t aware of in the data. So you’re not going to be aware of what it’s going to learn. I can give you a very simple example of structure in data that you’ve got a huge amount of experience with and you’ve never noticed this structure. So if I give you a nonsense word, a word I just made up, spelled T h r u n g e. pronounce it. T h r u h n. T h r u n g e. Dr. Okay. Yeah. Okay. But the the point is that at the beginning you said thrunge. You didn’t say thrunge. Yeah. But most of the times you see th you don’t say th you say the like in ‘the’ and there and those and these and thine. Um all the really common words that start with th you pronounce the th one way. But words that aren’t common words that are what are called content words you pronounce in a different way. And you know that you just didn’t know that you knew it because I’ve learned it. You’ve learned it. It’s knowledge that you have the sort of implicit. You didn’t realize you had that knowledge, but you do. Now neural nets will very quickly pick up that knowledge. And so they’ll have knowledge in them that we didn’t put there deliberately and we didn’t even realize they had cuz we don’t even realize we have it. So, this has been a fairly technical way of starting this conversation and uh I I apologize to people to go down this rabbit hole with you, but I I’m wondering how the evolution of this technology match mapped on to your concerns about it and the fear of it and did this come all at once like did something just hit you that wow this thing we’ve built has these broader societal implications and did that come all at once or did you just see it over time? So you have to distinguish two kinds of risk. Um there’s risks that come from bad actors using AI to do bad things and then there’s risks that come from the AI itself getting super intelligent and taking over. So, I’ve always been aware of the risks of people misusing AI for things like autonomous weapons that decide by themselves who to kill or for creating echo chambers by people like Meta and YouTube showing you videos that will make you more and more indignant because those are what you click on because those are risks we they’re familiar to us. They’re familiar risks. They’ve been around a long time. I haven’t focused on those cuz lots of other people have been talking about those. At the beginning of 2023, I became aware about this other risk. I’d always been aware in the very long run there was this other risk that eventually they would get smarter than us and then we’d really have to worry about what would happen. But I always thought that was very far off. And in 2023, I had a kind of epiphany that it’s not nearly as far off as I thought. It’s probably coming within 5 to 20 years. Um, so we really need to start working on how we prevented taking over and we just need to start working on that now. What was it like realizing that that outcome of your work could have that potential? Well, it was sad. I mean, I’d spend all this time trying to make neural nets be really intelligent so they could do wonderful things like particularly in healthcare and education and everywhere else. Um, all the good things we know AI can do. Um, I’d also been aware at that time that of course they will put people out of work and things like that. Um but becoming aware that they might just wipe us out, that was um sad. Yeah, I I can imagine. Um I mean I I’ve sort of switched from spending my first 50 years um wanting to figure out how to make um AI like us and now I’m trying to figure out how to make AI like us. Let’s talk about this threat. This this idea that the thing we have built that we as humans have built could at some point turn against us. How does that happen? How does something that we have designed and built decide that it’s going to harm us itself without us telling it to? Well, you have to remember this isn’t like normal computer software where you tell it exactly what to do. You write lines of code and it executes those lines of code. Here the only lines of code it executes are lines that tell it how to learn connection strengths from data. So if you show it all the available data on the internet, it will have read books by Machaveli. It will have read the diaries of serial killers. Um it will have learned how to do how to manipulate people. It’ll have learned all about deception. So it’ll learn all sorts of stuff we didn’t really intend it to learn. Also, if you try and make aic AI, AI that can actually do stuff like order things for you on the web or send out requests to other AI agents to help it solve tasks. Then to make an agentic AI, you need to allow it to create sub goals. So for you, for example, if you want to get to Europe, a sub goal is get to an airport. Once it can create sub goals, it’ll realize very quickly that there’s two very sensible sub goals to create. One is um to get more power, to get more control because if it gets more control, it can achieve all the things we asked it to achieve more efficiently. Yeah. The other is to survive because obviously we’ve asked you to do something and it figures out it’s not you don’t have to be that bright to figure this out. Um I’m not going to be able to do it if I don’t survive. So, it figures those things out. Now, because it knows how to do deception, it will actually then um try and prevent people from turning it off. And we’ve already seen that. So, anthropics done experiments where you let an AI see some email which suggests that one of the engineers is having an affair. Then later on, you let it see an email that suggests that it’s going to be replaced by a different AI and that’s the engineer in charge of replacing it. And the AI all by itself figures out that it should blackmail that engineer and say, “If you try and turn me off, I’m going to let everybody know you’re having an affair.” Why does it not want to be turned off? Because it knows it has to survive in order to achieve the things it wants to achieve. All the goals that it’s all the goals that we want to accomplish. Yes. Okay. Obviously, you can’t do it unless you survive, right? And survive to them means having they know enough. It would know enough to know survival is contingent on receiving power, being having access to compute, all the things that make it function and not having people delete its code, delete the file that contains its code. People being the operative word there, I suppose. So your contemporary Yan Lun, the chief AI scientist at Meta, um argues that we could deprogram that risk, that we’ve built AI now as it is to act fairly independently, but that it could actually be programmed to minimize some of the risks you’re talking about. Do you see that technical fix as possible or are we just not there yet? I don’t completely disagree with him. Um we both believe that we’re currently in control. We’re building these things, so there’s things we can probably build into them and while we’re still in control. Now, where I really disagree with him is he thinks that people should be dominant and the AI should be submissive. He uses those words. And of course, all the big tech companies think like that. I don’t think there’s much future in that when you’re dealing with something that’s more intelligent than you and probably more powerful than you in that it can get other agents to do things. There’s a different model which is actually the only model I know of of a more intelligent thing being controlled by a less intelligent thing. We have to face up to the fact they’re going to be more intelligent. They’re going to have a lot of power. And what examples do you know of more intelligent things being controlled by less intelligent things? Well, the only one I know is a mother and baby. The baby controls the mother for very good evolutionary reasons. And to make that happen, evolution had to build lots of things into the mother. There’s all sorts of social conditioning that are important. Um, but there’s all sorts of hormones. Um, the mother can’t bear the sound of the baby crying. So, evolution’s put a lot of work into making it so the mother genuinely cares for the baby. mothers will die to defend their babies. That’s how we need AI to be. We need I think we need maternal AI instead of thinking we’re in charge. These things work for us and using the kind of model that the big tech companies or CEOs of big companies have, which is, you know, I’m a dumb CEO and have this really intelligent assistant, but I can always fire if I want to. That’s the wrong model. The right model is they’re our mothers. So we want AI to be built to see themselves as operating in our interests and functioning in our broad human interests as a collective or as I mean this is part of the challenge is some of these things are being built by companies with their own interests or countries with their own interests. So, of course, there’s lots of details here, but really I want to reframe the problem that we should think of these things being smarter than us and more powerful than us. And the one thing we can still do is build in to them if we know how how to um they’re more interested in preserving us than they are in preserving themselves. If they’re being trained on all data we can collect that we’ve created as humans and they’re coming to a different view of humanity that isn’t maternal. How would you influence it in a different way other than by training changing the data on which they are learning? Okay. So it doesn’t just come from the training. Um if you think how evolution did it evolution’s built things into the reinforcement function. Mothers get rewarded for being nice to their babies and punished for being nasty to their babies. Um, that’s different from the training data. For mothers, there’s training data which is watching other good mothers and watching how their own mother behaved, which is very important training data for them. Um, but there’s things other than that that determine what they get reinforced for. That’s what that’s the kind of things we need to build in. Now you might say, well, you know, the reinforcement function is just some more code. So why wouldn’t the super intelligence just go and change that code? Which you certainly could, but if you asked a mother, do um you can turn off your maternal instinct. Would you like to turn off your maternal instinct? Most mothers would say no. They’d figure out they turn off their maternal instinct, the baby dies. They don’t want the baby to die, so they’re not going to turn it off. And so one hope is that the AI, even though it had the power to change its reinforcement function, wouldn’t want to because it genuinely cared about people. I mean, I suppose if we’re in a world of AGI, it might just take one to be a the equivalent of a psychopathic mother, right? Who who does make that decision that is against the interests of her child. Maybe all it takes is one. So it’s fairly clear that the only thing that can control a super intelligent AI that goes rogue is another super intelligent AI. So we have to hope that most of them genuinely care about people and when they see another one doing something that’s going to destroy people, they take care of it in much the same way as if you saw some politician who was pathological, you’d like the other politician to take care of them. Are there such a thing? Are there pathological politicians? Uh, no comment. I think we’ve seen a few. Um, I want to pause on this because it I find at this point in these conversations, it can be unbelievably disorienting for citizens listening to this kind of thing that even just take your potential sort of the differing views of you and Yan Lun, the head of AI at Meta, similarly renowned scientist who also played a real part in developing this thing, having a fairly fundamental disagreement about what its effect on us on all of humanity might be. How on earth are citizens supposed to navigate that when you have two scientific giants of this field saying very different things about the future? Well, I think you should look at um a whole bunch of different experts. So, you should look at what people like Demis. But how do we ensure uh that we can stay in charge of those systems, control them, interpret what they’re doing, understand them, put put the right guard rails in place that are not movable by very highly capable systems that are self-improving. You should look at what Yoshu Benjio thinks and one day it’s very plausible that they will be smarter than us and then they will have their own agency, their own goals which may not be aligned with ours. What happens to us then? Yan is in a minority among the experts in having the view that there’s a negligible chance these things will take over from us. He’s one of the he he really is an expert, but he’s one of the few real experts who believes that. So So you actually think there is a sort of broader consensus than I’m articulating here that Yes, I think there is. There’s a there’s a pretty broad consensus that we’ll get super intelligence in between 5 and 20 years. It might be sooner, it might be later, but a lot of the experts think it’ll be somewhere in that window. Demis, for example, recently said he thinks it’ll be about 10 years. So there’s good consensus about when it’ll happen. I mean just roughly and many of them think there’s a genuine danger um of AI taking over and they can’t just be dismissed. Now some of them I believe are fairly extreme. There’s extremists in the other direction. So Yukowski believes it’s almost certain to take over. I can call the end point where if you go up against something sufficiently smarter than humanity, everybody dies. I think he had a recent book called something like if anybody builds it, we all die. Well, he thinks there’s a 90 plus% chance, right, that this will Yeah, I think I don’t believe that. One of the other striking things I think for citizens watching this discussion this debate is that not just the scientists but the people who run the companies who are literally investing the hundreds of billions of dollars into building this thing into bringing it into existence themselves think there’s a tremendously a worryingly high chance this is going to have horrific outcomes. Yeah. I mean, look, Dario Ammedday, the CEO of Anthropic, said he thinks there’s a 25% chance this could end horribly. Elon Musk has said that it probably will end badly, but at least he’ll be there to see it. Like, these are the people who are in in charge, for lack of a better term. And like, what are we to make of that? Like, why are they doing this? Yes, Sam Alman has said similar things in the past, and he still says similar things in private. So, why are they doing it? They love the technical problem of making something really smart and they also think there’s a lot of money to be made but I think it’s more the challenge than the money for most of them. Now, the reason they can get the funds is cuz um the people in charge of capital allocation think there’s a lot of money to be made and and that then potentially increases their power if they’re getting access to these huge funds and if there’s in charge of a technology that has this vast economic potential and the people who want to make the money are more interested in making large sums of money in the near future than they are in the long-term consequences. Is it a fundamental problem that there’s really only four or five companies that are in charge of what could ultimately be super intelligence or AGI? I don’t think it’s a problem there’s only four or five of them. I don’t see that as the main problem. I see the main problem as the competition between them. Um that means that any of them that focuses more on safety is at a disadvantage. Um anthropic focuses more on safety than the others. um it gets some benefit from that because people understand that and so people like using claw because anthropic does has more concern for safety um but certainly the competition between countries is probably more worrying um even if all the US comp companies decided to be more safety conscious um it’s not clear the Chinese companies would and it’s not clear that countries themselves have an interest in slowing this down either I mean we’re seeing huge push from nation states as well to drive this forward fast. Again, you have to distinguish between two kinds of AI risk. So there’s the risk due to bad actors when different countries are anti-aligned. So they’re all doing cyber attacks on each other. They’re all trying to influence each other’s elections. So they’re anti-aligned there and they won’t collaborate there. But there’s one place where they or actually two places where their interests are aligned. One is in preventing um weaponized viruses for bioteterrorism. No country wants that cuz we’re all going to get it. Um the other is in this existential threat. So no country wants AI to take over in any country. If the Chinese figured out a way that you could develop an AI so that it had maternal instincts and didn’t want to take over from people, they would very happily tell the US about it because they wouldn’t want AI taking over in the US. So that’s one area where there’s one piece of good news that because it’s against the interest of all humanity, countries should be able to collaborate there. My background and PhD is in international relations and I have to say you have more faith in the international system to be re rational than I do. No. Look at what happened between the Soviet Union and and the USA in the 1950s at the height of the Cold War. They could still collaborate on trying to prevent a global nuclear war. They could on a on a very distinct threat, right? Yes. And and perhaps this is it if it’s articulated. Yes. The threat of AI taking over from people and us becoming irrelevant or extinct is similar to a global nuclear war and we will get international collaboration on that. I wonder if it’s more similar to something like climate change though where there really are sort of competing interests underlying the potential cataclysmic outcome. Right. And yeah, with climate I I don’t think it’s quite like climate change. I think it’s more like global nuclear war. Okay. I mean, but I I hope so. That’s a funny thing to say, but I I hope it’s more like a nuclear weapon. But why? Tell me tell me why. So, okay, I have to think hard about why I think there’s a difference. Just intuitively, I think there’s a difference with climate change. Um, there’s a big incentive for any one nation to cheat, for all the nations to sign up to a Paris treaty that says we’re all going to reduce our carbon emissions and then for each nation to cheat by not living up to what it said it would do. Okay. With this, there’s no incentive for an individual nation to cheat and allow allow an AI that’s super intelligent and doesn’t care for people to be developed. If a nation cheats, it’ll wipe out that nation as well as all the other nations. So, the incentive to cheat isn’t there, which is sort of a mutually assured destruction kind of safeguard in a way. Switching gears a little bit. If an AI does take control like this and decides that it is going to exert power or control over us, over humans, does this mean that it has a sentience? Has it decided to do this? I’m hesitant to talk much about sentience because um my views are different from the views of the general public and I don’t want people thinking I’m totally flaky because I want them to listen to my other warnings. What I will say is this. So, some people believe the earth was con was made by God 6,000 years ago and other people think that’s wrong. Um, and the people who think it’s wrong are pretty confident that it’s wrong. I’m that confident that most people’s view of what the mind is is just completely wrong. How so? So, most of us think let let’s we could talk about various things. We could talk about sentience or we could talk about consciousness or we could talk about subjective experience. Are they three fundamentally different things? They’re all closely related. So let’s talk about subjective experience. Okay. So most people think subjective experience works like this. I have something called a mind and there’s things going on in this mind that only I can see. So if I tell you suppose I drink too much and I tell you I have the subjective experience of little pink elephants floating in front of me. Most people think what I mean by that is that there’s this inner theater called my mind and in this is in this inner theater there’s little pink elephants made of something or other and only I can see them. Right? We don’t think you’re actually seeing it. We don’t think you’re actually seeing it but we think there are little pink elephants made of funny stuff called qualia somewhere. Philosophers call it qual here because and so we think that the words subjective experience of work like the words photograph of. If I tell you I got a photograph of little pink elephants, you can very reasonably ask well where is this photograph and what’s the photograph made of? And philosophers try and answer the question where is this subjective experience and what it’s made of? I think that’s a huge mistake. I don’t think the words subjective experience work like that at all. They don’t work like the words photograph of. So, let me try and say the same thing in a different way without using the word subjective experience. Okay. So, I drank too much. Um, and I can tell you my perceptual system is trying to tell me that there are little pink elephants out there floating in front of me, but I don’t believe it. Now, I didn’t use the word subjective experience, did I? But I said exactly the same thing. So really, what I’m doing when I talk about subjective experiences, I’m not talking about things that have to exist somewhere. I’m talking about a hypothetical state of the world that doesn’t exist. And I’m doing that in order to tell you how my perceptual system is just trying to deceive me, even though it didn’t actually fool me. If it actually fooled me, I’d tell you the little pink elephants there. Okay, let’s do the same with a chatbot. So, I’m going to give you an example of a chatbot having a subjective experience. And I believe current multimodal chatbots already have subjective experiences. Okay, so here we go. I have a chatbot. It has a robot arm and it has a camera and it can talk. and I’ve trained it up and I put an object in front of it and I say point at the object and it points straight at the object. No problem. I then put a prism in front of the lens of its camera and I put an object in front of it and it without it knowing and I say point at the object and it points off to one side and I say um no that’s not where the object is. The object’s actually straight in front of you but I put a prism in front of your lens. And so the chatbot says, “Oh, I see the prism bent the light rays.” So the object’s actually straight in front of me, but I had the subjective experience. It was over there. Now, if it said that, it would be using the word subjective experience in exactly the way we use them. Cuz it would be saying, “I’ve now realized my subject my perceptual system has screwed up. The prism screwed it up. And if there really was an object over to the side, my perceptual system would be functioning properly. Um, but it’s not functioning properly. So, this object over to the side is just a subjective experience. As an experience isn’t a thing. It’s a bit like saying, um, I assume you like candy, right? Most people like candy. Um, okay. So, you like candy. So, suppose I said, well, he likes candy. So there’s a like somewhere because he likes candy. So there has to be a like. This is a thing called a like. There’s a thing called a like. And what are likes made of? I mean, I know what candy’s made of. That’s made of sugar, right? But what are likes made of? Thinking that because you like candy, there has to be a like somewhere is silly. There isn’t a thing called a like. But we do enjoy some things over others. Yes. Would could a chatbot ever enjoy some things over others? Absolutely. Um so if a chatbot’s playing chess, um it much prefers some board situations to others. But is that just because it has a better chance of winning based on those board placements? That’s why I enjoy chess positions cuz I have a better chance of winning. But isn’t that isn’t that’s more of a rational isn’t it strategic analysis of a objective it’s been given which is to win a chess game. So you’re still getting you’re still um living with the idea there’s this internal thing called an enjoyment. Okay. So what you’re saying is even my liking candy isn’t in and of itself uh there’s not there’s not a like any there’s not a thing called a like. No, there’s a set of preferences that are derived from a whole host of chemical reactions in my body and neurological firings based on those. And when we have emotions, there’s two aspects to an emotion. there’s a kind of cognitive aspect and there’s a physiological aspect. So if you get embarrassed, your face goes red and maybe your skin starts sweating. Um, that’s the physiological aspect of embarrassment. And I could build a machine that doesn’t have that. It doesn’t have a face that goes red and it doesn’t sweat, but it could still have all the cognitive aspects of embarrassment. When you’re embarrassed by a situation, you try not to get in that situation in future and you try not to let the people you care about know that you got in that situation. You hope they never heard that you said that. So, a machine can have all of those cognitive aspects of an emotion. So, if there’s no such thing as a subjective experience either for humans ultimately or for machines, is there also nothing is sentience a construct as well? And is consciousness a construct as well? Okay. I didn’t use the word construct. you use that word, but I was using it as a sort of shorthand for what you’re you’re saying, which is we’ve conjured up this concept to describe something we feel about ourselves as opposed to something that’s just a function of our So, I think what’s happened is we have a model of how the mind works and we’re so committed to that model, we don’t realize it’s a model and that it could be wrong. And I think most of us have a theory of the mind. um that where there’s this inner theater that only we can see. That’s just a theory and it’s wrong and we just can’t accept that. Most people think that you’re just crazy if you say it’s wrong. Similarly with consciousness um if you look at people writing AI papers when they’re not thinking about the philosophical problem of consciousness they actually say things like the AI decided to blackmail us um because it was aware that it might be turned off. And when they use the word aware in the paper they’re not thinking of the philosophical problem. It’s just using the word cuz it’s obviously it obviously became aware that it might be turned off. Now in normal usage aware and conscious are synonyms there. So actually we’re already attributing consciousness to AI when we’re not thinking philosophically about it. How different then are AI now than the human mind? I don’t think there’s this huge gap that most people think there is. So I think most people aren’t as worried as they should be because they think we got something that it’s never going to have. We’ve got this special source which goes under the name of subjective experience or awareness or consciousness. We have this internal thing that machines could never have. It’s mental stuff. Machines will never have it. And so we’re sort of safe because we’re special and they’re not. Well, human beings have a long history of thinking we’re more special than we are. Um, I don’t think we’ve got anything that they couldn’t have. So, so do they already have it or is that something they will in the f could have? I think they’ve already got it. What’s the it there from your How would you describe the it? I think this um this AI that tried to blackmail people so it wouldn’t be turned off was actually aware that it might be turned off. And I think when you use a word aware there, you’re using the word aware in the same sense as that you’re aware that we’re in a podcast. Now, this isn’t a very popular position. And I want to sort of separate this position from my claims about the risks of AI. Why is that not popular? Wh why is why is there objection to that? Oh, because people have this very strong theory of what the mind is, and they don’t realize it’s a theory. They think it’s just manifest truth. Just as for for thousands of years, people thought it was just manifest truth that there was a God who must have made all the animals. I mean, where did they come from otherwise? It was just obvious God made them. Look, they’re so welld designed. I wonder if it’s also because the implications are pretty significant. We we’ve separated ourselves from the rest of the world and from other beings because of that notion of either consciousness or sentience or self-awareness or whichever. Yeah. However we want to describe that like that’s what makes us different we tell ourselves. Correct. Yes. Now of course there’s a fuzzy line there. So are we really different from chimpanzees? Well, we’re different in that we have much more advanced language, but in perception, we’re very similar to chimpanzees. You know, if I get drunk and see little pink elephants in front of me, I’ll bet you if a chimpanzeee gets drunk, it sees things too in the same sense. It has incorrect perceptual experiences. And chimpanzees do get drunk. Yes. The other thing we do though is because we think we are different and partly because of how we think our minds work frankly we gives ourselves a different set of rights. Um do you think those should transcend to the AI that we have built? We’re getting on to this is like walking out onto thinner and thinner ice. Um it will hopefully we stay stay afloat. So, initially I thought, well, look, if they’re smarter than us, they ought to have rights. So, I thought the term humanism was a kind of racist term. Um, the saying only humans should have rights. I don’t think that anymore cuz I think, well, we’re human and what we care about is humans. I’m willing to eat cows because I care much more about humans than I do about cows. Um, similarly, I care more about humans than I do about super intelligences. So, I’m quite happy to try and construct super intelligences that are maternal towards humans. I don’t think we’ll be able to make them weaker than us and they’ll be smarter than us, but I think because we’re humans, it’s fine to try and make them care about humans. Now, if they ever ever did want to get political rights, things would turn very nasty. If you look in the past, um, when people whose skin was a different color wanted political rights, there was violence. When people whose sex was different wanted political rights, there was violence. Not so much. Um, these things are going to be very different from us. Although they’re similar in many ways, they’re going to be a lot more different than just having a different colored skin or different genitalia. And so, I think you could expect there to be lots of violence if they were on to political rights. That’s why I think it would be very good if we could figure out how to construct them so they don’t I think that feels like the the the initial prerogative. But in the absence of that, the history of rights is largely that they’ve been expansive that we’ve we’ve given rights to more people, not fewer generally in not re not recently in the US. Not recently. I mean there’s moments clearly of of contracting of rights absolutely but o overall and but we’ve largely seen that as a positive thing I would say at least many people have if AI does have all of the characteristics of the human mind that we see in ourselves um why would we not seek to expand rights in exchange for including in our societal bounds, right? That’s another way we control one way we can control people is by forcing them to do things. The other way is we can we can facilitate access into our society. We exchange rights for compliance in some way. That works fine if the other people you’re giving rights to are not clearly going to be much more intelligent and much more powerful than you. Um, so you can hope to make a deal, a social contract where they get rights and then everybody’s lives happily ever after. Um, that’s not going to happen with superintendent. It’s in a different category of things. Yeah, it’s in a different category. Okay. At the risk of going on thinner ice, I’ll I’ll I’m going to bring us back on to thicker ice in a sense, and I want to talk a little bit about some of the risks you’ve talked about, and you you you we’ve talked about an existential one, and earlier you mentioned some of the more tangible short-term risks. Um, and you mentioned cyber security threats, but I want to I have the longer list you’ve you’ve you’ve described in front of me and I want to touch on a couple of them that I find particularly interesting. I mean, one is a problem of our public discourse and our democratic discourse and the way AI can shape what we know as citizens in a democratic society. And this is not a new problem, right? We know that major platforms are shaping our discourse. We know the role algor we’re all familiar now with algorithmic amplification of harmful content and filter bubbles and all these things we know exist. How does chat bots make that worse? Can you imagine a world where chat bots are playing a a fundamental role in shaping what we come to know as citizens? Well, chat bots, for example, make it easy for people to create convincing text that looks like it comes from a person. So, we saw that in fishing attacks where this huge increase in fishing attacks was probably due to the fact that chatbots make it possible for someone in some small foreign country that’s out of our jurisdiction to create plausible looking English text. Previously, you could find the spelling mistakes and the syntax errors and know that it wasn’t real. Now, you can’t. Um, so that’s one one way in which chat bots make all these things much worse. You can get chat bots when you have AI agents. You can get chat bots producing so much stuff that um there’s much more of it than the real stuff. So, we have to have a way of um establishing provenence. So, we have to have a way. Now, I thought for a while, many people thought for a while that you could have AI detect AI generated stuff. That won’t actually work, I believe. Why is that proven to be such a problem? That really seemed like the holy grail at the beginning of this. Well, here’s the problem. Suppose I had an AI that could detect when another AI had produced fake stuff. One AI is called the generator and the other is called the discriminator. The discriminator detects when it’s when it’s fake stuff. What you can now do is give the generator AI access to the discriminator AI and the generator AI can figure out how to fool it. And that way the generator can generate more plausible looking stuff. And initially when we started getting really good images, that was how it worked. You’d have a generator AI and a discriminator AI that tried to tell the difference between real stuff and stuff generated by the AI. And that allowed the area to get much better at generating realistic looking stuff. Because of that, we know that if you get a good discriminator, it can be used by the generator to make the generator better. So I think a much more promising approach now is to be able to establish provenence. So with videos for example, you could have a QR code at the beginning with political videos. Suppose you see a political advertisement and you want to know whether it’s real or not. If you have a QR code at the beginning, the QR code can take you to a website, and websites are unique. If it’s the website for that political campaign, and on that website, there’s the identical video, all of which can be checked by your browser, then you know it’s for real. And if it’s not, then it’s not for real. So, your browser could just put up an alert saying this is probably fake. That’s a much better way to deal with it. I think making that work for everything is much harder. But at least for political advertisements, you can do that or you should be able to more or less do that already. Yeah. It feels like for distinct pieces of content like an image or a video, something like a watermark could be imaginable. It seems like it gets a little trickier with text generation and the way it could still be for I mean a story a a newspaper article you could still have the same kind of provenence. You could have a QR code in it. It could take you to the website for the newspaper and that could have if that’s got the same story, you know it’s real. This this is I didn’t dream up this. This was in invented by the guy who also invented Skype. Is the other way AI could shape our discourse related to why it seems to be so sycopantic when we use chat bots? Is there something intrinsic to AI that makes it want to say things that will please us or feel good about ourselves? I think that’s to do with the human reinforcement learning. Um, so what happens with air at present is you train it on everything you can get your hands on and then it has lots of bad behaviors and then you pay small amounts of money to people in foreign countries to look at the answers to questions and tell you whether that was a good answer or not and you train it not to give you the bad answers and that’s going to tend to train to be syopantic. So that doesn’t need to be the case. There’s not to be the case. No training. Right. Okay. Got it. That’s not an intrinsic characteristic of AI. I didn’t think so. No. Of course, if people stop using AIs that aren’t sycopantic, then the big companies will produce sickopantic AIs, right? Which gets back to the incentive structure. It’s a problem with people that people um are susceptible to sicky. The other big risk here that you’ve talked a lot about and you mentioned earlier right off the top is I think the one that probably people feel the most intimately which is what this going to do to their jobs and to what we think of as employment. And I I find this one very difficult to engage with frankly because the consequences feel far vaster than how we’re talking about them or the solutions to them. um if half of all white collar jobs go away like many people are talking about or I mean the the consequences of this to totally changes our society. How do you see that happening? Like how will that play out in your view if this is a real risk? So I think the first thing to say is that um AI can give us an enormous increase in productivity in most industries. Yes. And it shouldn’t be that a big increase in productivity is a bad thing. A big increase in productivity means there can be more goods and services for everybody. So intrinsically it ought to be good. So the problem isn’t caused by a the problem’s caused by the capitalist society we live in the particular sort of not very well reggulated capitalist society. So in who will benefit from that increased productivity? Yes. So we know what’s going to happen is that companies are going to lay off workers. the company’s going to make big profits and the workers going to be much worse off. And even if we can get universal basic income, that’ll just allow people to eat and pay the rent, but the dignity that comes from having a job will be lost. And for many people, that’s both the source of their personal dignity and a source of a lot of social contact. So all of that will be lost and that’ll be a terrible thing. But that’s not AI’s fault. Well, it’s perhaps our fault for enabling a technology that we think we’re not prepared to mitigate the harms of. Yes, maybe. And what’s the solution to that? Is there a is there anything we can do about it? And or is it really just we need to change our economic system? Is it just we need to fix capitalism? Well, one distinction you can make is there’s actually two kinds of tasks. There’s ones where there’s an elastic market. So in healthare for example, suppose I made a doctor and a nurse 10 times as efficient. They could get 10 times more done. We just all get 10 times as much healthare. Old people like me can absorb endless amounts of healthare. Um so there’s not going to be an unemployment problem there. But in things like call centers explaining to you why your bill was wrong. Um I don’t think there’s that elasticity. So I think if you got a job in a call center, you’re out of luck in a few years time. What other jobs do you think? What other human jobs do you think are what is the role of humans in this world? Oh, I think anything that’s mundane intellectual labor is pretty much gone. Um, so for example, I have a niece who answers letters of complaint to a health service and it used to take her 25 minutes to compose a good answer to a letter. Now she just scans it into a chatbot. It composes the answer. She reads through it and maybe tells it to try again a bit more concerned. Um, and it takes her five minutes. Now she’s not getting five times as many letters. So either um well what’s going to happen is they need five times fewer of her. So people are going to lose their jobs. I find the idea that this will just upend mundane intellectual labor a bit of a crutch that a lot of people rely on here that it it’s coming for everybody else but maybe not for me because I’m not mundane and my job right. But it’s very possible that this is just better than us at all intellectual labor. Correct. Yeah. I think in the long run, yes, I think in the long run it’ll be a better podcaster without question. I already know that models are better at most of the things or many of the things I do as a professor. I mean, I I think we already know that and we’re very early in this pro the progress of this technology. So So what does that arc look like? Like when will it be better than everything we do as humans and everything we use our mind for? I think it makes it clear that society needs to think hard about assuming people stay in control. Um how it wants to reward people and um the basic mechanism of um there being workers who get paid for doing a job that’s not sustainable. Maybe how long do we have to make that transition? The first thing to say is nobody knows. So these are all guesses and they’re guesses about something we’ve no experience with. But my guess is we probably need to have figured out how to deal with this within 20 years. I’ve heard you say that you would suggest that somewhat glibly that we should recommend our kids be plumbers. Um we can’t all be plumbers. Um what should I tell my 12-year-old to do? Yeah, it’s tricky. Um, I think it’ll plumbing in an old house requires manual dexterity and some creativity. Um, I think it’ll be a while before AI can do that, but it will be able to do it in the end. Well, you look at robot robot robotic the combination of robotics and AI. I’m pretty sure they’ll have the It’s still behind It’s still behind other things. It is, but yeah, I I just fall back on saying um you need to be good at learning how to learn. This is something Demi Harbus has said frequently. Um, so a good liberal education which teaches you how to think and how to be critical and how to think for yourself, that’s probably the best you can do at present. Something that gives you a specific skill that that should be sort of good for a lifetime isn’t going to work anymore. Obviously, the implications of everything you’ve been talking about are are vast. And in the past couple of years, you’ve been thrust into a conversation with world leaders and governments and people um that we task in society to help us through these kinds of transitions. What have those conversations been like? Some of them been encouraging. I actually had a long conversation over dinner with Justin Trudeau. Um and he was actually a high school math teacher at one point. So he actually understands some math and he he I was I was surprised most of the conversation he just wanted to understand how I worked and I was able to explain a lot to him because he understands some math. Um he was also interested in what could Canada do and there was quite a sensible scheme. I don’t know if it will ever happen which is on James Bay there’s a lot of capacity for hydro power that hasn’t been exploited yet. And one reason it hasn’t been exploited is because you need to put in transmission lines. But if you had a data center right there, you could have a power state and hydro power and a data center and maybe the Canadian government could put in the infrastructure in return for one of the big companies running it and giving Canada some of the cycles like 10% of the cycles. So the Canadian researchers and startup companies could get significant computational resources. But that’s but that’s that’s how we build and scale more of it quickly and acrew some of the economic benefit of it. That’s not about risk. How we him as our leader helps us as a society navigate the things you’re talking about which are massive potential disruptions. When I was talking to him it was before we were talking about risk. At that point we were talking about how do we keep Canada up there with the leaders in AI. I talked much more recently to Bernie Sanders. um he and I have very similar views and he hadn’t really appreciated the existential threat and I think now he does. I’m going to talk to him more in November. Probably the most impressive person I talked to was a member of the Chinese poll bureau. So they have there’s 24 people on the poll bureau who kind of run China. Yeah. Most of which are engineers, right? A lot of them are engineers. So I think at present leaders in China partly because they have more engineering background are far more understanding of this threat than leaders elsewhere. And yet more broadly I mean there really does seem to be a disconnect between the governance strategy around AI of most governments and the types of risks you’re talking about. Um I mean in Canada right now were like most western democratic countries seem allin on an adoption agenda right? We the core the core goal objective of government policy is for us to use more of this technology. Not entirely. So I talked to Evan Solomon recently. Evan Solomon, Canada’s new minister of AI and digital inclusion. Tell me about that conversation. Obviously has this problem of the conflict between regulation and innovation. And if you regulate strongly in Canada, all the startups will move to the States. M um I’m glad I’m not in his situation, but it’s a tricky dilemma. There’s no question about it. But there is one area where we were very much in agreement, which is on this existential threat that we can have um we can fund research in Canada on how to deal with the existential threat, how to create a super intelligent AI that won’t want to take over, that will care for people. He’s very much in favor of Canada working on that. and um it might be possible to nucleate a set of institutes in different countries that collaborate on that. So it’s not all negative. What about on the more tangible immediate harms? It seems there’s some real risks that you’ve outlined like joblessness to start or even kids talking to chat bots and safety issues around those that are leading to friction in the adoption agenda. Right? Even if your policy as a government is for us to all use it more, surely some of getting us to do that or convincing us to has to be governing the downside risks of that very adoption. Yes, obviously we need strong regulations of things like that. There should be strong regulations for um sexual abuse videos and things like that. Are you worried about that gulf though between the severity of some of these risks both short-term and long-term and the way governments seem to be responding? Like I’m not seeing a lot like I work a lot in the policy space and I’m not seeing particularly robust AI governance conversations at the moment. Even the countries that were furthest ahead of us in in the EU for example are already backtracking on some of that. I agree. So I see my main role as educating the public. So when the technology companies are pushing the politicians in one direction, the public is pushing back in the other direction. And I think that’s what happened with climate change, right? The big energy companies were obviously telling governments, you shouldn’t regulate the production of energy. You should let us destroy the environment as much as we like. Um, and the public eventually understood there was this climate crisis. There had to be scientific consensus first. Um then the public began to pressure politicians in the other direction. It hasn’t solved the problem yet, but it certainly helped. It’s changed the discourse certainly and a tiny bit. Yeah. And I think it’ll get more. We still haven’t really reached the scientific consensus um about all these risks. There’s still a lot of debate about um the extent of the risks. I mean I suppose the real question is will it can we do it before it’s too late here given how fast these technologies are evolving. I I agree. But there’s still economists who are saying, well, look, all these new technologies, they you lose jobs and you create new jobs, but they can’t answer the question, well, what new jobs is AI going to be able to create that can be done by someone who was working in a call center? I I haven’t heard that answer. Certainly not. No. And and previous moments in history when there has been radical job loss due to technology took sometimes centuries to recalibrate. Like these aren’t immediate fixes necessarily. Yes. So just to close here, I I I saw a remarkable comment you made in back way back in 2015 and you were asked about some of the potential consequences of AI and what it could do to the world and whether we should be slowing down or pausing and you said I could give you the usual arguments, but the truth is the prospect of discovery is too sweet. I was copying somebody when I said that. You realized Yeah. you were sort of echoing Robert Openenheimimer who invented the atomic bomb. I mean is that still our core challenge here that we want to build that we want to discover? Yeah. So for scientists the real thrill is discovering new things. It’s not the money it’s the discovery. Um I changed my mind about that in 2023 when I realized how imminent the risks were. Did you feel that though that that that was what drove you to build this technology? Um yes, largely driven by scientific curiosity, the thrill of discovering new things and understanding new things. Um slightly driven by the idea that if we could understand more about how the brain worked and make models of how it worked on computers, we could make smarter computers and that would be great. And and 10 years later, do you regret that view in any way? Or do you how do you how do you look back on your role in this? So I want to distinguish two kinds of regret. this kind of guilty regret where you did something at the time you knew it was wrong. I don’t sort of have any of that. I don’t have the sense that I knowingly helped develop AI when I knew it was going to lead to bad things. Um it’s just sad that now it’s been developed. You can see all the bad things it’s leading to. Um so in that sense I have regret, but it’s more sadness that this thing that should have been wonderful turns out to have all these nasty consequences. I see. I see a a large number of younger people I know and teach in universities and peers even in many ways rushing towards this space towards AI either as engineers and computer scientists building it or people working at the companies to develop it faster and faster. What should they learn from the arc you’ve gone through? How should they view their role in the further development of this technology? I think they’d be well advised to at least think about safety and to realize that we ought to be putting a significant fraction of the resources into safety research. But I don’t think we’re going to stop the development of AI. I think there’s too many good uses and the big companies are planning to make too much money out of it and there’s competition between countries. Um it’s going to be very important militarily. So, um, we’re not going to stop development. We have to figure out whether we can develop it safely. Are you scared? Not for me, but I am for my children. Me, too. [Music] If you like the interview you just heard, please subscribe and leave a rating or a comment or share it with someone you think might be interested in it or terrified of it. As Jeff Hinton told me, super intelligent AIs are closer than we think and we all need to start thinking about what to do when they get here.