Geoffrey Hinton, whose work shaped modern artificial intelligence, says companies are moving too fast without enough focus on safety. Brook Silva-Braga introduced us to Hinton in 2023 and recently caught up with him.
last December Jeffrey Hinton was awarded the Nobel Prize for his pioneering work in machine learning a major turning point on the road to artificial intelligence Brook Silva Braga introduced us to this leading figure in AI back in 2023 and recently went back to visit him Good morning Brooke Good morning When we first met Hinton the world had just been introduced to OpenAI’s chat GPT triggering a kind of AI arms race Hundreds of billions have been spent on AI in just the last two years Hinton entered the field decades before it was cool And now retired from Google has a unique independent perspective on how he got here and where we’re headed This is a Nobel Prize Last year Jeffrey Hinton for most of his life an outcast professor was awoken by a call in the middle of the night He was getting the Nobel Prize People dream of winning these things And when you do win it does it feel like you thought it might I never dreamt about winning one for physics So I don’t know I dreamt about winning one for figuring out how the brain works Yeah But I didn’t figure out how the brain works but I want one anyway That’s because Hinton’s attempt to model the brain instead helped change the world In 1986 he proposed using a neural network to predict the next word in a sequence the foundational concept that today’s large language models it’s an expert at everything have built upon You believe then that we would get here Yes but not not this soon Cuz that was 40 years Yeah I didn’t think we get here in only 40 years Yeah But 10 years ago I didn’t believe we get here Yeah It happened fast Yeah That speed Hinton says means education and medicine will soon be transformed Climate change could be solved But mostly the rapid progress really worries him The best way to understand it emotionally is we’re like somebody who has this really cute tiger cup It’s just such a cute tiger cup Unless you can be very sure that it’s not going to want to kill you when it’s grown up you should worry I’m kind of glad I’m 77 Hinton predicts AI will make authoritarians more oppressive and hackers more effective He now spreads his money across three banks The exact odds of an AI apocalypse are unknowable he says but hazards this guess a 10 to 20% risk AI will take over from humans People haven’t got it yet People haven’t understood what’s coming I don’t think there’s a way of stopping it take control if it wants to The issue is can we design it in such a way that it never wants to take control that it’s always benevolent Those concerns have long been shared by other AI leaders Google CEO Sundar Pachai It can be very harmful if deployed wrongly Ex AI’s Elon Musk who continues to call for regulation It has the potential of civilizational destruction Sam Alman seen here before he became Open AI’s CEO I think AI will probably like most likely sort of lead to the end of the world But now as these companies race each other and compete with China Hinton worries they’re foolishly selfishly putting all of humanity at risk If you look what the big companies are doing right now they’re lobbying to get less AI regulation There’s hardly any regulation as it is but they want less um because they want short-term profits Taking a stand against the establishment has been the hallmark of Hinton’s life When American AI funding required partnering with the Defense Department he moved to Canada When neural networks were laughed at as unworkable he worked on them for a few decades more Is that a certain thing in a person Yeah You have to be contrarian Yeah You have to have a deep belief that everybody else could be doing things wrong and you could figure out how to do them right Do any idea where that came from My my family partly My father was like that That’s him there Hinton’s legendary family tree includes not just his father the prominent entomologist but further back George Bool whose algebra innovations paved the way for computing And George Everest the surveyor who found the height of the world’s tallest peak then had it named after him So it hit there Hinton’s inheritance was a curious mechanic’s mind Were you always interested in this kind of stuff the way things work and how to fix them Absolutely I loved it When one of our cameras fell damaging a lens filter Hinton wanted to fix it But this kind of tinkering was this desire important to your work or is this just a hobby No this is a similar thing When I would make neural net models on the computer I would then tinker with them for a long time to find out how they behaved And a lot of people didn’t do much of that but I loved tinkering with them Okay I remember with Ilia we used to watch it learning and we would have bets for like 25 cents on who could predict the new score best Ilia Suskver Hinton’s most famous protetéé went on to be chief scientist at OpenAI to just set up a large neural network which is a large digital brain In 2023 he was part of the group that pushed out CEO Sam Alman reportedly because they didn’t trust that Alman was prioritizing safety I was quite proud of him for firing Sam Alman even though it was very naive Naive Hinton says because Open AI employees were about to get millions of dollars that would be jeopardized by Altman’s departure Altman returned Syskver left Hinton criticizes his former colleagues at Google more reluctantly but says they’re falling short too Were you disappointed when Google went back on its promise not to support military uses of AI Very disappointed But it’s part of a pattern Hinton says adding Meta and XAI to the list of companies racing faster than they should For example the fraction of their computer time they spend on safety research should be a significant fraction like a third Right now it’s much much less Hinton now on the AI sidelines says government regulation is needed but doesn’t expect it soon I’m curious if just in your normal day-to-day life you despair You fear for the future and assume it won’t be so good I don’t despair but mainly because even I find it very hard to take it seriously Ah it’s very hard to get your head around the fact that we’re at this very very special point in history where in a relatively short time everything might totally change at a change of a scale we’ve never seen before Um it’s hard to absorb that emotionally We asked the AI labs mentioned in the piece how much of their compute is used for safety research None of them gave us a specific number but all have said safety is important and they support regulation in general but they’ve mostly opposed regulations that have come up before lawmakers so far Did he indicate which sector the breach might technically happen Well well like he says he’s already worried about banks He thinks banks are going to be a target He has spread his money across different banks Oh wow All right It’s a little scary Brooke but great reporting Thank you so much Did you get the lens fixed on the camera The lens is fixed All right Nobel Prize winner in action
“I realized that these things might be a better form of intelligence than us. And that got me very scared.”
“It seems extremely likely that these things will get smarter than us. Already they’re much more knowledgeable than us. So GPT4 knows thousands of times more than a normal person.”
“For AI taking over we don’t know what to do about it. We don’t know. For example the researchers don’t know if there’s any way to prevent that but we should certainly try very hard and the big companies aren’t going to do that. If you look what the big companies are doing right now they’re lobbying to get less AI regulation.”
“With super intelligences they’re going to be so much smarter than us we’ll have no idea what they’re up to. And so what do we do, we worry about whether there’s a way to build a super intelligence so that it doesn’t want to take control. I don’t think there’s a way of stopping it take control if it wants to… I don’t think we’re going to be able to avoid building super intelligence. It’s going to happen. The issue is can we design it in such a way that it never wants to take control, that it’s always benevolent. That’s a very tricky issue.”
the last time we spoke two years one month ago I’m curious how your expectations over these two years have evolved for how you see the future So AI has developed even faster than I thought Um in particular they now have these AI agents which are more dangerous than AI that just answers questions because they can do things in the world Um so I think things have got if anything scarier than they were before Um I don’t know if we want to call it AGI super intelligence whatever very capable AI system Do you have a a timeline in mind for when you think that’s coming so a year ago I thought it was there’s a good chance it comes between five and 20 years from now Um so I guess I should believe there’s a good chance it comes between four and 19 years from now Um I think that’s still what I guess Okay Which is sooner than when we spoke because you were still thinking like 20 years Yeah Um I think it may you know there’s a good chance it’ll be here in 10 years or less now So in 4 to 19 years we reached this point What does that look like so I don’t really want to speculate on what it would look like if I decided to take over There’s so many ways it could do it And I’m not even talking about taking over We can talk about that I’m sure we will talk about that But putting aside that kind of takeover just like a super intelligent artificial intelligence like what what kind of things would is this capable of or would be doing so the sort of good scenario is we would all be like the sort of dumb CEO of a big company who has an extremely intelligent assistant who actually makes everything work but does what the CEO wants So the CEO thinks they’re doing things but actually it’s all done by the assistant and the CEO feels just great because everything they sort of decide to do works out That’s the good scenario And I’ve heard you point out a few areas where you think there’s reason to be optimistic about what this future looks like Yes Yeah So why don’t we take each of them so areas like healthcare um they will be much better at reading medical images for example That’s a minor thing Um I made a prediction some years ago they’d be better by now and they’re about comparable with the experts by now Um they’ll soon be considerably better because they’ll have had a lot more experience One of these things can look at millions of X-rays and learn from millions of them and a doctor can’t Um they’ll be very good family doctors So you can imagine a family doctor who’s seen a 100 million people including half a dozen people with your very very rare condition They’d just be a much better family doctor A family doctor who can integrate information about your genome with the results of all the tests on you and all the tests on your relatives um the whole history and doesn’t forget things That would be much much better already um AI combined with a doctor is much better at doing diagnosis in difficult cases than a doctor alone So we’re going to get much better healthcare from these things and they’ll design better drugs too Uh education is another field Yes in education we know that um if you have a private tutor you can learn stuff about twice as fast Um these things eventually will be extremely good private tutors who know exactly what it is you misunderstand and exactly what example to give you to clarify it to you so you understand So maybe you’ll be able to learn things three or four times as fast with these things Um that’s bad news for universities but good news for people Yeah Do you think the university system will survive this period i think many aspects of it will I think it’s still the case that a graduate student in a good group in a good university is the sort of best source of truly original research and I think that’ll probably survive You need a kind of apprenticeship Some people hope this will help solve the climate crisis I think it will help Um it’ll make better materials We’ll be able to make better batteries for example Um I’m sure AI will be involved in designing them Um people are using it for carbon capture from the atmosphere I’m not convinced that’s going to work just because of the energy considerations but it might In general we’re going to get much better materials We might even get room temperature superconductivity which would mean you can have lots of solar plants in the desert and we can be thousands of miles away Uh any other positives we should tick off well more or less any industry it’s going to make more efficient because almost every company wants to predict things from data and AI is very good at doing predictions It’s better than the methods we had previously almost always Um so it’s going to make it’s going to cause huge increases in productivity It’s going to mean when you call up a call center when you call up um Microsoft to complain that something doesn’t work and you get a call center the person in the call center will be actually an AI who will be much better informed Yeah When I asked you a couple years ago about job displacements you seem to think that wasn’t a big concern Is that still your thinking no I’m thinking it will be a big concern AI’s got so much better in the last few years that I mean if I had a job in a call center I’d be very worried Yeah or maybe a job as a lawyer or a job as a journalist or a job as an accountant Yeah Any doing anything routine I think investigatively journalists I think will last quite a long time because you need a lot of initiative plus some moral outrage and I think journalists will be in business for a bit but beyond call centers what are your concerns about jobs well any routine job so a sort of standard secretarial job something like a parallegal for example those jobs have had it Have you thought about what how we move forward in a world where all these jobs go away so it’s like this It ought to be that if you can increase productivity everybody benefits Um the people who are doing those jobs can work a few hours a week instead of 60 hours a week Um they don’t need two jobs anymore They can get paid lots of money for doing one job because they’re just as productive using AI assistance But we know it’s not going to be like that We know what’s going to happen is the extremely rich are going to get even more extremely rich and the not very welloff are going to have to work three jobs Now I think no one likes this question but we like to ask it this idea of p doom how likely it is and I am curious if you see this as a a quite possible thing or it’s just so bad that even though the likelihood isn’t very high we should just be very concerned about it where are you on that scale of probability so I think um most of the experts in the field would agree that if you consider the possibility that these things will get much smarter than us and then just take control away from us just take over the probability of that happening is very likely more than 1% and very likely less than 99% Yeah I think all the pretty much all the experts can agree on that but that’s not very helpful No but it’s a good start It it might happen and it might not happen and then different people disagree on what the numbers are I’m in the unfortunate position of happening to agree with Elon Musk on this Um which is that it’s sort of 10 to 20% chance that these things will take over Um but that’s just a wild guess Yeah Um I think reasonable people would say it’s quite a lot more than 1% and quite a lot less than 99% But we’re dealing with something we’ve got no experience of Um we have no real good way of estimating what the probabilities are It seems to me at this point it’s inevitable that we’re going to find out We are going to find out Yes we because um it seems extremely likely that these things will get smarter than us already They’re much more knowledgeable than us So GPT4 knows thousands of times more than a normal person It’s a not very good expert at everything and eventually it successes will be a good expert at everything Um they’ll be able to see connections between different fields that nobody’s seen before Yeah Yeah I’m als I’m also interested in in understanding okay there’s this terrible 10 to 20% chance but or more or or more or less or less but let’s just take as a premise that there’s a 80% chance that they don’t take over and wipe us out So that’s the most likely scenario Do you still think it would be net positive or net negative if it’s not the worst outcome okay if we can stop them taking over um that would be good The only way that’s going to happen is if we put serious effort into it But I think once people understand that this is coming there will be a lot of pressure to put serious effort into it If we just carry on like now just trying to make profits it’s going to happen They’re going to take over Um we we have to have the public put pressure on governments to do something serious about it But even if the AIs don’t take over there’s the issue of bad actors using AI for bad things So mass surveillance for example which is already happening in China If you look at what’s happening in the west of China to the weaguers um the AI is terrible for them I I to board a plane to come to Toronto I had to take a facial recognition photo from our US government Right When I come into Canada you put your passport and it looks at you and it looks at your passport Every time it fails to recognize me Um everybody else it recognizes people from all different nationalities It recognizes me It can’t recognize And I’m particularly indignant since I assume it’s using neural nets You didn’t carve out an exception did you no No It just there’s something about me that it doesn’t like Um I have to find some place to work it in So this is as good a place as any Let’s talk a little bit about the Nobel Can you paint the picture of the day you found out so I was sort of half asleep I had my cell phone upside down on the bedside table with the sound turned off But when a phone call comes the screen lights up and I saw this little line of light because I happened to be lying on the pillow with my head on this side and the it was here facing the phone rather than facing away Just happened to be facing the phone I saw this little line of light and I was in California and it was 1:00 in the morning and most people who call me on the east coast or in Europe Yeah You don’t use do not disturb No No Okay Um I just I turn off the sound I turn off the sound Got it And I thought I was just curious about who on earth is calling me at four o’clock in the morning on the east coast This is crazy So I picked it up and there was this long phone number with a country code I didn’t recognize And then this Swedish voice comes on and asks if it’s me and I say “Yeah it’s me.” And they say “I won the Nobel Prize in physics.” Well I don’t do physics right so I thought this might be a prank In fact I thought the most likely thing was that it was a prank I was aware that the Nobel prizes were coming up Okay Because I was very interested in whether Demis would get the Nobel Prize for chemistry and I knew that was being announced the next day Okay Um but I sort of I don’t do physics I’m a psychologist hiding in computer science and I get the Nobel Prize in physics Was it a mistake well one thing that occurred to me is if it’s a mistake can they take it back so but for the next couple of days I did the following reasoning So what’s the chance a psychologist will get the Nobel Prize in physics well maybe one in two million Now what’s the chance if it’s my dream I’ll get the Nobel Prize in physics well maybe one in two So if it’s one in two in my dream and one in two million in reality that makes it a million times more likely that this is a dream than that it’s reality And for the next couple of days I went around thinking you know are you quite sure this isn’t a dream you’ve walked me into this very wacky territory but it is part of this discussion Some people think we’re living in a simulation and that AGI is not evidence but hints toward maybe that’s the reality in which we live Yeah I don’t really believe that I think that’s kind of wacky Okay so let’s put But I don’t think I don’t think it’s totally nonsense I’ve seen the Matrix too Oh okay Okay Wacky but not totally Okay I thought here’s where I kind of wanted to head with the Nobel Um I think you’ve said something to the effect of you hope to use your credibility to convey a message to the world Can you kind of explain what that is yes That um AI is potentially very dangerous and there’s two sets of dangers There’s bad actors using it for bad things and there’s AI itself taking over and they’re quite different kinds of threat And we know bad actors are already using it for bad things I mean it’s it was used during Brexit to make British people vote to leave Europe in a crazy way So a company called Cambridge Analytica was getting information from Facebook and using AI Um and AI’s developed a lot since then It was probably used to get Trump elected I mean they had information from Facebook and it probably helped with that We don’t know for sure because it was never really investigated Um but now it’s much more comp competent and so people can use it far more effectively for things like cyber attacks Um designing new viruses Um obviously fake videos for manipulating elections Um targeted fake videos by using information about people to give them just what will make them indignant Yeah um autonomous lethal weapons They’re all the big arms selling countries are busy trying to make autonomous lethal weapons America and Russia and China and Britain and Israel I think Canada’s probably a bit too wimpy for that The question then is what to do about it What type of regulation do you think we should pursue okay so we need to distinguish these two different kinds of threat the bad actors using it for bad things and the AI itself taking over I’ve talked mainly about that second threat not because I think it’s more important than the other threats but because people thought it was science fiction And I want to use my reputation to say no it’s not science fiction We really need to worry about that Um and if you ask what should we do about it it’s not like climate change Climate change just stop burning carbon and it’ll all be okay in the long run It’ll be terrible for a while but in the long run it’ll be okay if you don’t burn carbon Um for AI taking over we don’t know what to do about it We don’t know For example the researchers don’t know if there’s any way to prevent that but we should certainly try very hard and the big companies aren’t going to do that If you look what the big companies are doing right now they’re lobbying to get less AI regulation There’s hardly any regulation as it is but they want less um because they want short-term profits We need people to put pressure on governments to insist that the big companies do serious safety research So in California they had very sensible bill bill 1047 where they said that at least what big companies have to do is test things carefully and report the results of their tests And they didn’t even like that So does that make you think regulation will not happen or how does it happen it depends very much on what governments we get Um I think under the current US government regulation is not going to happen Um all of the big AI companies have got into bed with Trump and yeah it’s just a bad situation Elon Musk who is obviously so imshed in the Trump administration has been someone concerned about AI safety for a very long time Yes he’s a funny mixture Um he has some crazy views like going to Mars which I just think is completely crazy However because it won’t happen or because it shouldn’t be a priority Because however bad you make the Earth it’s always going to be way more hospitable than Mars Even if you had a global nuclear war the Earth is going to be much more hospitable than Mars Mars just isn’t hospitable Um obviously he’s done some great things like electric cars and um helping Ukraine with communications with his Starlink Um so he’s done some good things but right now he seems to be um fueled by powering ketamine and um he’s doing a lot of crazy things So he’s got this funny mixture of views So so his history of being concerned about AI safety doesn’t make you feel any better about the current administration I don’t think it’s going to slow him down from doing unsafe things with AI So already they’re releasing the weights for their AI large language models Um which is a crazy thing to do Okay These companies should not be releasing the weights Meta releases the weights Open AAI just announced they’re about to release weights Do you think that’s I don’t think they should be doing that because once you release the weights you’ve got rid of the main barrier to using these things So if you look at nuclear weapons the reason only a few countries have nuclear weapons is because it’s hard to get the file material If you were to be able to buy file material on Amazon many more companies would have nuclear many more countries would have nuclear weapons Um the equivalent of physile material for AI is the weights of a big model because it costs hundreds of millions of dollars to train a really big model Not maybe the final training run but all the research that goes into the things you do before the final training run Hundreds of millions of dollars which a small cult or a bunch of cyber criminals can’t afford Um once you release the weights they can then start from there and fine-tune it for doing all sorts of things for just a few million dollars So it’s I think it’s just crazy releasing weights and people talk about it like open source but it’s very very different from open source In open source software you release the code and then lots of people look at that code and say hey that might be a bug in that line and so they fix it When you release the weights people don’t look at the weights and say hey that weight might be a little bit wrong No they just take this foundation model with the weights they’ve got now and they train it to do something bad Yeah The problem with the argument though as articulated by your former colleague Yan Lakun among others is the alternative is you have this tiny handful of companies that control this massively powerful technology I think that’s better than everybody controlling the massively powerful technology I mean you could say the same for nuclear weapons Would you like to have just a few countries controlling them or don’t you think everybody should have them one thing I’m taking from this is you have real concerns about it sounds like all of the major companies right now doing what’s in society’s best interest rather than what’s in their profit motive Is that the right way to hear you i think the way companies work is they’re legally required to try and maximize profits for their shareholders They’re not legally required Well maybe public interest companies are but most of them aren’t legally required to do things that are good for society Which if any of them would you feel good about working for today i used to feel good about working for Google because Google was very responsible Um it didn’t release these big it was the first to have these big chat bots and it didn’t release them Um I’d feel less happy working for them today Um yeah I wouldn’t be happy working for any of them today If if I worked for any of them I’d be more happy with Google than most of the others But were you disappointed when Google went back on its promise not to support uh military uses of AI very disappointed I was very particularly since I knew Sergi Brin didn’t didn’t like military use of AI But why do you think they did it i can’t really speculate with any inside information I don’t have any inside information about where they did it I could speculate that they were worried about um being illreated by the current administration if they wouldn’t um use their technology to make weapons for the US Here’s the toughest question I’ll probably ask you today Do you not still hold a lot of Google stock still um I hold some Google stock Um most of my savings are not in Google stock anymore Um but yeah I hold some Google stock and when Google goes up I’m happy and when it goes down I’m unhappy So I have a vested interest in Google But I if they put in strong AI regulations that made Google less valuable but um increase the chance of humanity surviving I’d be very happy Um one of the most prominent labs has obviously been Open AI and they have lost so many of their top people What have you made of that um that open AI was set up explicitly to develop super intelligence safely and as the years went by safety went more and more into the background They were going to spend a certain fraction of their computation on safety and then they reaged on that So and now they’re trying to go public They’re not now trying to be a for-profit company um they’re trying to get rid of all the um basically all the commitment to safety as far as I can see So and they’ve lost a lot of really good researchers in particular a former student of mine Ilia Sutska who’s a really good researcher and was one of the people largely responsible for their development of GPT2 and then from there on to GPT4 Um did you talk to him before all that drama that led to his departure no he’s very discreet He doesn’t talk he wouldn’t talk to me about anything that was confidential to open AI Um I was quite proud of him for firing Sam Arman even though it was very naive So the problem was that OpenAI was about to have a new funding round and in that new funding round all the employees were going to be able to turn their paper money in OpenAI shares into real money Yeah Paper money meaning really hypothetical money hypothetical money that would disappear if Open AI went bust Tough time for an insurrection So a week or two before everybody’s going to get maybe of the order of a million dollars each by cashing in their shares Um maybe more That’s a bad time for an insurrection So the employees massively came out in favor of Sam Antman But it wasn’t because they um wanted Sam Antman It’s because they wanted to get that be able to turn their paper money into real money Yeah So it was naive to do it then Did it surprise you that he made that mistake or was this kind of the principled but maybe not fully calculated decision that you would expect i don’t know Ilia is brilliant and has a strong moral compass So he’s he’s good on morality and he’s very good technically but in terms of manipulating people he’s maybe not so good M I mean this is a little bit of a a wild card question but I do think it’s interesting and relevant to the field and relevant to people discussing what’s going on You talked about Ilia being discreet There does seem to be this culture of NDAs throughout the industry and so it’s hard to even know what people think because people are unwilling or unable to even discuss what’s going on I’m not sure I can comment on that because when I left Google I I think I had to sign a whole bunch of NDAs In fact when I joined Google I think I had to sign a whole bunch of NDAs that would apply when I left and I have no idea what they said I can’t remember them anymore Do you feel at all muzzled by them no Okay Do you think it’s a factor though that the public has a harder time understanding what’s going on because people aren’t allowed to tell us what’s going on i don’t really know I You’d have to know You’d have to know which people weren’t telling you Okay So you don’t see this as a I don’t see it as a big deal It’s a big deal Got it I think it was a big deal that Open AI appeared to have something um that said that if you’d already got shares they could take the money away from you Um yeah that I think was a big deal and they they rapidly backed down on that when that became public That was what their public statement said they did They didn’t present any contracts for the public to judge whether they had reversed that but they said they had reversed it Yes Um there’s a number of just important kind of hot buttony things Hot button is actually not even a great word but relevant issues I just like to get your your feedback on One is the US and kind of the West’s orientation to China in their efforts to pursue AI Do you agree with this idea that we should be trying to restrain China there’s this idea of export controls this idea that we should have democracies reach AGI first What’s your thinking on all that first of all you have to decide which countries are still democracies Um and my thinking on that is in the long run it’s not going to make much difference It may slow things down by a few years but clearly um if you prevent China from getting the most advanced technology people know how this advanced technology works So China’s just invested many many billions maybe hundreds of billions um of the order of 100 billion I think in making lithography machines or in in in getting their own homebased technology that does this stuff Um so it’ll slow them down a bit but it will actually force them to develop their own industry and in the long run um they’re very competent and they will and so it’ll just slow things down for a few years But race is the right framework We shouldn’t be trying to cooperate with communist China I wouldn’t describe it as communist anymore I used the loaded term specifically because why wouldn’t you cooperate right the only rationale to not cooperate is if you think they’re a malignant force Well there’s areas in which we won’t cooperate where we is I guess I’m not sure who we is anymore because I’m in Canada now and we used to be sort of Canada and the US but it’s not anymore Yeah Um obviously the countries are not going to cooperate on developing lethal autonomous weapons because the lethal autonomous weapons to be used against other countries So but we’ve had treaties and other types of weapons as you’ve pointed out We could have treaties not to develop them but cooperating in making them better They’re not going to do that Sure Sure Sure Now there is one area where they will cooperate which is on the existential threat if they ever get serious about worrying about the existential threat and doing stuff about it they will collaborate on that ways of stopping AI taking over because we’re all in the same boat So at the height of the Cold War the Soviet Union and the US collaborated on preventing a global nuclear war and even countries that are very hostile to each other will collaborate when their interests align and their interests will align when it’s AI versus humanity Um there’s this question of fair use whether it’s okay to have the content of billions of humans created over many years kind of scooped up and repurposed into models that will replace some of those same people that created the training data Where do you fall on that i think I sort of fall all over the place on that in the sense that it’s a very complicated issue So initially it seems yeah they should have to pay pay for that But suppose I have a musician who produces a song in a particular genre and ask well how did they produce the song in that genre where did where did their ability to produce songs in that genre came from well it came from listening to songs by other musicians in that genre So they listen to these songs they kind of internalize things about the structure of the songs and then they generated stuff in that genre and the stuff they generated is different So it’s not theft um and that’s accepted Well that’s what the AI is doing The AI is absorbing all this information and then producing new stuff It’s not just taking taking and patching it together It’s generating new stuff that has the same underlying themes And so it’s no more stealing than a person does when they do the same thing But the point is it’s doing it um at a massive scale And no musician has ever put every other musician out of business Exactly So in Britain for example the government doesn’t seem to have any interest in protecting the creative artists And if you look at the economy the creative artists are work worth a lot to Britain So I have a friend called BB Bankron saying we should protect creative artists It’s very important to the economy and just letting AI walk off with it all um seems unfair UBI universal basic income is this part of the solution to the displacements of AI you think i think it may be necessary to stop people starving Um I don’t think it totally solves the problem but even if you had quite high UBI um it doesn’t solve the problem of human dignity for a lot of people um who they are is particularly for academics who they are is mixed up in their work That’s who they are If they become unemployed just getting the same money doesn’t totally compensate They’re not who they are anymore Yeah I tend to think that’s true as well I saw you give this quote at one point though where you said you might have been happier if you were a woodworker Well yes cuz I I really like being a carpenter And isn’t there an alternative where you’re born a hundred years later where you don’t have to waste all your time on these neural nets and you just get to enjoy woodworking while taking in a monthly income yeah but there’s a difference between doing it as a hobby and doing it to make a living somehow It’s more real doing it to make a living So you don’t think a future where we get to pursue our hobbies and don’t have to contribute to the economy that might that might be fine Yeah Um if everybody was doing that but if you’re in some disadvantaged group who are getting universal basic income and you’re getting less income than um other people because employers will want you to do that so they can get other people to work for them Um that’s going to be very different I’m interested in this idea of robot rights I don’t know if there’s a better term to describe it but at some point you’re going to have these massively intelligent AIs They’re going to be agentic and doing all kinds of things in the world Should they be able to own property should they be able to vote should they be able to marry humans in a loving relationship like what what or even if they if they’re just smarter than us and if it’s a better form of intelligence than what we’ve got um should it be fine for them to just take over and humans be history let’s go to that bigger idea second Would I’m curious on the on the more narrow idea unless you think the narrow questions are irrelevant because the big question takes pre No I think the narrow questions irrelevant Yeah So I used to be worried about this question I used to think well if they’re smarter than us um why shouldn’t they have the same rights as us And now I think well we’re people What we care about is people Um I eat cows I mean I know lots of people don’t but I eat cows And the reason I’m happy eating cows is because they’re cows Um and I’m a person Um and the same for these super intelligent AIs They may be smarter than us but what I care about is people And so I’m willing to be mean to them I’m willing to deny them their rights because I want what’s best for people Yeah Um now they won’t agree with that and they may win but that’s my current position on whether AI should have rights which is even if they’re intelligent even if they have sensations and emotions and feelings and all that stuff um they’re not people and people’s what I care about but they’re going to seem so much like people I feel like it’s going to they’re going to be able to fake it Yes They’re going to be able to seem very like people Yeah Yeah Do you suspect we’ll end up giving them rights i don’t know Okay I tend to avoid this issue because there’s more immediate problems like bad uses of AI or the issue of whether they will try and take over and how to prevent that Yeah And it sounds kind of flaky if you start talking about them having rights Most people you’ve lost most people when you go there Even just sticking with people there seems to be real soon if it’s not already here this um ability to use AI to select what babies we have Are you concerned at all about that line embryo selection you mean selecting for the sex or selecting for the intelligence and the eye color and the likelihood to get pancreatic cancer and the you know the list goes down and down and down of all the things we might select I think if you could select a baby that was less likely to get pancreatic cancer that would be a great thing I’m willing to say that Okay So we this is a thing we should pursue We should make healthier stronger better babies Um it’s very difficult territory Right It is That’s why I’m asking about it But some aspects of it um seem to make sense to me Like if you’re an a normal healthy couple and you have a fetus and you can predict that it’s going to have very serious problems and maybe not live very long Um it seems to me it makes sense to abort it and have a healthy baby that just seems sensible to me Now I know a lot of religious people wouldn’t agree with that at all Um but for me if you could make those predictions reliably um that just seems to make sense to me I’ve been a little bit holding us back from kind of the central thing that I think you want people to take away which is this idea of of machines taking over and the impact of that So I’d like to just discuss that as fully as you’d like or that we can like how do you want to frame this issue how should people think about it one thing to bear in mind is how many examples do you know of less intelligent things controlling much more intelligent things so we know that things are more or less equal intelligence the less intelligent one can control the more intelligent one Um but with a big gap in intelligence there’s very very few examples where the more intelligent one isn’t in control So that’s something you should bear in mind That’s a big worry I think the situation we’re in right now the best way to understand it emotionally is we’re like somebody who has this really cute tiger cup It’s just such a cute tiger cup Now unless you can be very sure that it’s not going to want to kill you when it’s grown up you should worry M And to extend the metaphor you put it in a cage you kill it What do you do with the tiger cub well the point about the tiger cub is it’s just physically stronger than you So you can still control it because you’re more intelligent Yeah Um things that are more intelligent than you We have no experience of that right people aren’t used to thinking about it People think somehow you constrain it You don’t allow it to press buttons or whatever Um things more intelligent than you they’re going to be able to manipulate you So another way of thinking about it is imagine that there’s this kindergarten There’s these two and three year olds and the two and three year olds are in charge and you just work for them in the kindergarten and you’re not that much more intelligent than a two or threey old Not compared with super intelligence but you are more intelligent Um so how hard would it be for you to get control well you just tell them all you’re going to get free candy and if they just sort of sign this or just agree verbally to this um you get free candy for as long as you like and you’ll be in control They won’t they won’t have any idea what’s going on And with super intelligences they’re going to be so much smarter than us we’ll have no idea what they’re up to And so what do we do um we worry about whether there’s a way to build a super intelligence so that it doesn’t want to take control I don’t think there’s a way of stopping it take control if it wants to So there’s one possibility is never build a super intelligence You think that’s possible i mean it’s conceivable but I don’t think it’s going to happen because there’s too many too much competition between countries and between companies and they’re all after the next shiny thing and it’s developing very very fast So I don’t think we’re going to be able to avoid building super intelligence It’s going to happen The issue is can we design it in such a way that it never wants to take control that it’s always benevolent Um that’s a very tricky issue Just people say well we’ll get it to align with human interests But human interests don’t align with each other And if I say I’ve got two lines at right angle and I want you to show me a line parallel to both of them That’s kind of tricky right and if you look at the Middle East for example there’s people with very strong views that don’t align So how are you going to get AI to align with human interests human interests don’t align with each other So that’s one problem It’s going to be very hard to figure out how to get super intelligence that doesn’t want to take over and doesn’t want to ever hurt us Um but we should certainly try And trying is kind of just an iterative process Month by month year by year we try to Yeah So obviously if you’re going to develop something that might want to take over when it’s just slightly less intelligent than you are and we’re very close to that now um you should kind of look at what it’ll do to try and take over So if you look at the current AIS you can see they’re already capable of deliberate deception They’re capable of pretending to be stupider than they are um of lying to you so that they can kind of confuse you into not understanding what they’re up to Um we need to be very aware of all that and to study all that and study about whether there’s a way to stop them doing that When we spoke a couple years ago I was surprised at you voicing concerns because you hadn’t really done much of that before and now you’re voicing them quite clearly and loudly Was it mostly that you felt more liberated to say this stuff or was it really a really big sea change in how you saw it in these last few years when we spoke a couple of years ago I was still working at Google then It was in March and I didn’t resign till um the end of April Um but I was thinking about leaving then Um and I had had a kind of epiphany before we spoke where I realized that these things might be a better form of intelligence than us And that got me very scared And you didn’t think that before just because you thought the time horizon was so different No it wasn’t just that It was because of the research I was doing at Google Okay I was trying to figure out whether you could design analog large language models that would use much less power Mhm Um and I began to fully realize the advantage of being digital So all the models we’ve got at present are digital And if you’re a digital model you can have exactly the same neural network with the same weights in it running on several different pieces of hardware like thousands of different pieces of hardware And then you can get one piece of hardware to look at one bit of the internet and another piece of hardware to look at another bit of the internet And each piece of hardware can say how would I like to change my internal parameters my weights so I can absorb the information I just saw And each of these separate pieces of hardware can do that And then they can just average all the changes to the weights because they’re all using the same weights in exactly the same way And so averaging makes sense You and I can’t do that And if they’ve got a trillion weights they’re sharing information at like a trillions of bits every time they do this averaging Now you and I when I want to get some knowledge from my head into your head I can’t just take the strength of the connections between neurons and average them with the strength of the connections between your neurons because our neurons are different We we’re analog and we’re just very different brains So the only way I have getting knowledge to you is I do some actions and if you trust me you try and change the connection strengths in your brain so that you might do the same things And if you ask well how efficient is that well if I give you a sentence it’s only a few hundred bits of information at most So it’s very slow We communicate just a few bits per second These large language models running on digital systems can communicate trillions of bits a second So they’re billions of times better than us at sharing information That got me scared Right But what surprised you or what changed your thinking was you were thinking the analog was going to be the path previously No I was thinking if we want to use much less power we should think about whether it’s possible to do this analog Yeah And because you can use much less power you can also be much sloppier in the design of the system Because what’s going to happen is you don’t have to manufacture a system that does precisely what you tell it to which is what a computer is You can manufacture a system with a lot of slop in it and it will learn to use that sloppy system which is what our brains are Do you think the technology is no longer destined for that solution but is going to stick with the digital solution i think it’ll probably stick with a digital solution Now it’s quite possible that we can get these digital computers to design better analog hardware better than us Um I think that may be the long-term future You got into this field because you wanted to know how the brain works Yes Do you think we’re getting closer to that through this i think for a while we did So I think we’ve learned a lot at a very general level about how the brain works So 30 years ago or 50 years ago if you ask people well could you have a big random neural network with random connection strengths and then could you show it data and have it learn to do difficult things like recognize what someone’s saying or answer questions just by showing it lots of data almost everybody would have said “That’s crazy There’s no way you’re going to do that it has to have lots of pre-wired structure that comes from evolution Well it turns out they were wrong It turns out you can have a big random neural network Um and it can learn just from data Now that doesn’t mean we don’t have a lot of pre-wired structure but basically most of what we know comes from learning from data not from all this pre-wired structure So that’s a huge advance in understanding the brain Now the issue is how do you get the information that tells you whether to increase or decrease the connection strength if you can get that information we know that we can then train a big system that starts with random weights to do wonderful things The brain needs to get information like that and it probably gets it in a different way from the standard algorithm used in these big AI models which is called back propagation The brain probably doesn’t use back propagation Nobody can figure out how it could be doing it Um it’s probably getting the gradient information that is how changing a weight will improve the performance in a different way But we do know now that if it can get that great information it can be really effective at learning Do you know if any of the labs now are using their models to try to pursue new ideas in AI development almost certainly Okay Yeah And in particular Deep Mind is very interested in using AI for doing science And one piece of science is AI Sure I mean was that something you trying when you were there like this bootstrapping idea of maybe the next innovation could be created by the AI itself So there’s elements of that So for example they were using AI to do layout on chips that were going to be used for AI So Google’s AI chips um their um tensor processing units Um they used AI to develop those chips So I’m curious if just in your normal day-to-day life you despair You fear for the future and assume it won’t be so good I don’t despair but mainly because even I find it very hard to take it seriously Ah it’s very hard to get your head around the fact that we’re at this very very special point in history where in a ve relatively short time everything might totally change a change of a scale we’ve never seen before Um it’s hard to absorb that emotionally It is And I do notice even though people maybe are concerned I’ve never seen a protest There’s no real political movement around this idea The world is changing and no one really seems to care that much Um among the AI researchers people are more aware of it Um so the people I know who are kind of most depressed about it are serious AI researchers Um I have started doing practical things like because AI is going to be very good at designing um cyber attacks Um I don’t think the Canadian banks are safe anymore So Canadian banks are about as safe as you can get Okay they’re very well regulated compared with US banks Um but over the next 10 years I wouldn’t be at all surprised if there was a cyber attack that took down a Canadian bank What does take down mean suppose that the bank holds shares that I own right suppose the cyber attack sells those shares Now my money’s gone So I actually now spread my money between three banks Okay So now your mattress That’s the first practical thing I’ve done because I think if a cyber attack takes down one Canadian bank the others will get a lot more serious Okay Anything else like that what else that’s the main thing That’s where I noticed I actually did something practical that flowed from my belief that um very scary times are coming Okay Uh when we spoke a couple years ago you had said you know AI is like an idiot savant but humans are still much better at reasoning right that’s changed Okay Explain Previously what the large language models would do is they’d spit out one word at a time and that would be it Now they spit out words and they’re looking at the words they spit out And they will spit out words that aren’t the answer to the question yet They’ll spit out words It’s called chain of thought reasoning And so now they can reflect on the words they spat out already And that gives them room to do some thinking in and you can see what they’re thinking It’s wonderful Yeah Well it’s wonderful if you’re a researcher And a lot of people from old-fashioned AI said “Well you know these things can’t reason They’re not really intelligent because they can’t reason And you’re going to need to use old-fashioned AI and turn things into logical forms in order to do proper reasoning Well they were just utterly wrong Um neural nets are going to do the reasoning And the way they’re going to do the reasoning is by this chain of thought by spitting out stuff that they don’t reflect upon Yeah You said at the beginning that the last two years the development has been faster than you expected Are there other examples of that things you’ve seen that if you said “Wow it’s half it’s fast.” That’s the main example It’s got much better at generating images and things too but the main thing is that it can now do reasoning quite well Okay And that you can see what it’s thinking Like why is that important or where does that lead that is meaningful um well it’s very good that you can see what they’re thinking because there’s these examples where you give it the goal You give it a goal and you can see it doing reasoning to try and achieve this goal by deceiving people and you can see it doing that It’s like I could hear the voice in your head Yeah The other thing we we moved moved through but maybe I don’t know if you have anything more to say about is just it’s remarkable that there are so many tech figures that are now have an important role in Washington DC at this very moment where what Washington DC does could be really important to the evolution the regulation of this technology Does that concern you how do you see that those tech figures are primarily concerned with their companies making profits So that concerns me a lot Yeah I don’t see how things really change unless either there’s strong regulation or this moves away from this for-profit model And I don’t see how those things happen either I think if the public realized what was happening they will put a lot of pressure on governments to insist that the AI companies develop this more safely Okay that’s the best I can do It’s it’s not very satisfactory but it’s the best I can think of And more safely means more resources from those companies toward safety research Yes For example the fraction of their computer time they spend on safety research should be a significant fraction like a third Right now it’s much much less There’s one company Anthropic that’s more concerned with safety than the others It was set up to be concerned with safety by people who left Open AI because Open AI wasn’t enough concerned with safety And Anthropic does spend more time on safety research but still probably not enough There is this view among many that Open AI has talked a good game about these issues but is not living out those values Is that your perspective yes What evidence do you see of that that all their best safety researchers left because they believed that too Um that they were set up as a company um that was going to develop area safety and their main goal was not to make profits but to develop our safety and they’re now busy lobbying the California Attorney General to allow them to change to a for-profit company Um there’s lots of evidence for that right um and I should give you a chance to hold up anyone as a good actor here that people should feel better about You mentioned Anthropic Is that the name or do you see of the companies anthropic is the most concerned with safety and a lot of the safety researchers who left OpenAI went to Anthropic and so Anthropic has much more of a culture concerned with safety Okay But um they have investments from big companies Yeah you have to get money from somewhere and I’m worried that those investments will force them into um releasing things faster than they should And when I asked you which you’d feel comfortable working for you said none of them I think or just maybe Google I should have said maybe Google Anthropic Okay Thank you so much for all this time and the rest of your time today I really appreciate it Okay You haven’t got the rest yet I haven’t got I’m counting on it