NTD’s Lee Hall sits down with AI safety expert Professor Roman Yampolskiy, who says we are only a couple of years away from AI superintelligence that will endanger humanity. Yampolskiy says the companies developing AI are aware of the danger, but are driven by technological competition and profit. He says we have no way to predict or control intelligence that is beyond our own.
so there is this game theoretic problem between different labs where they all want the government to step in make it illegal to develop super intelligence but they want to be the ones with the most advanced system at the time it happens So they all continue development until everyone else stops Coming up on British thought leaders Roman Yonolski says we’re only a couple of years away from AI super intelligence that will endanger humanity and that the companies behind this technology know how dangerous it is All of them are on record as saying there is a good chance it will create existential problem for humanity But once they ended up in this position where they are leading a for-profit organization all of a sudden they learn to ignore their own concerns Remen says leaders in tech and AI are speaking out to stop AI development before it’s too late Well I don’t really care about career moves I want to say what is actually true scientifically And at the time there was very few people who even looked at the problem Today we have Nobel Prize winners during award winners thousands of computer scientists all agreeing with me So it is not a French position That is a majority [Music] position Welcome to British thought leaders I’m Lee Hall Today I’m sitting down remotely with AI safety expert and author Professor Roman Yolski Roman thanks for joining us Thanks for inviting me A couple of years ago you and hundreds of other leading tech figures signed this open letter calling for a pause to training powerful AI systems The letter said they pose profound risks to humanity What are those risks what are the risks from developing super intelligent systems we don’t know how to control We don’t understand how they work and we have no idea what they’re going to do to us It’s very hard to predict They’re smarter than us I can tell you what a psychopath can do a crazy person but we really can’t specifically guess what a much more intelligent agent would be capable of doing I mean where are we on the timeline to creating this AGI super intelligence that’s beyond us if we listen to the leaders of top AI labs they’re telling us that within a year or two they’re going to get to AGI We’re getting similar numbers from prediction markets So it would be very surprising if within 3 to 5 years we didn’t have something like that So I mean the key problems that we don’t really know what it’s going to do and we can’t control it basically Right So there is a few things we would like to be able to do and that’s kind of part of my research Uh can we explain how the systems actually learn and perform not just in general sense but in terms of specifics Can we predict what a smarter agent would do can we control it either in terms of direct control or in terms of delegated control for all those uh there are well-known limits to what is possible So have there been examples of AI systems going a bit rogue doing something unexpected so relative to the capability of a system there are hundreds of examples of failures The system designed to do X usually fails to X So the trivial example would be you have a spell checker in your phone it autocorrected to the wrong definition to the wrong word something you dictate it all the time And we’re kind of used to that Your spam uh catching agent in your email folder redirects important email to a junk folder So those are trivial things We’re all used to them but they are problematic within the context of capabilities of that agent Now what if your agent is capable of doing a lot more and it fails in a similar way so there’s this term called PDoom which is the probability of existentially catastrophic outcomes from AI And once we have this AGI super intelligence what do you see as the probability of doom and how long would it take right So the important part is how long if you ask for a very shortterm interval what is the pedum for next month it’s probably not that high But if we keep developing new systems we’re constantly releasing new models You have your GPT5 coming out then 6 7 24 uh it’s an ongoing process Essentially we need to always be couple steps ahead We always need to make sure there are no problems no accidents it becomes a perpetual safety problem kind of like a perpetual motion device Uh I don’t think we can do it indefinitely So the longer you wait the higher your pedom gets eventually hitting very close to a guaranteed event So what would we say say 10 years 20 years 50 years what would that value be i think it’s approaching 100% over those time frames really So pretty much definitely we would get wiped out by this super intelligence unless something new comes out where we have a reason to think we can control those systems and so far no one has mathematical proof paper not even a blog post suggesting they know how to indefinitely control smarter than human systems So I mean other experts have put this pedium value at 10% 20% 50% What would you say to people who who accuse you of fear-mongering i I think it’s a realistic estimate of how dangerous this technology is There is not enough fear if anything You bring up example of people who have 10% as a pedum That’s insanely high That is betting all of humanity and one in 10 chance of us being destroyed so a corporation can increase profits That is unacceptable So would you say people working in AI are putting being technological leaders making progress or even profit above the safety of others uh definitely we see it all the time If we look at their public statements before they became CEOs before they became leaders of those labs they all acknowledged that AI safety is a big problem They had very significant concerns about AGI and super intelligence All of them are on record as saying there is a good chance it will create existential problem for humanity But once they ended up in this position where they are leading a forprofit organization all of a sudden they learn to ignore their own concerns As a member of the public I kind of imagine these people working in AI are regulated there’s a legal framework they’re working within and they wouldn’t really do things that would cause massive danger Is that just a a naive outlook so first there is absolutely no regulation for what you allow to do on your computer in terms of programming You can program any agent you want and no one will stop you Uh if that was not the case and some laws were passed limiting what is possible to do we have no capability to meaningfully monitor or test those models It takes years to fully understand what a model is capable of We still discover new capabilities and new bugs in old models released a long time ago There’s very little media talk about this The public don’t seem to worried about it Why is there no concern well there is a lot of talk about it a lot of podcasts and interviews And the public is actually a lot more concerned than the experts If experts have 10% concern public is well over 50% and most prefer that we don’t develop super intelligence systems to replace them Do you think it’s a case of for for some people it’s not kind of on our doorstep So we’re not too worried at the moment and maybe by the time people are very concerned it would be too late So we are talking about within a year or two that is literally maybe too late at this point to do anything The problem is for most average people there is not much they can do Even if they completely accept this reality what are they going to do to make a difference so how can we stop this disaster from happening if we had some self-interest most of us would not try to develop a system which will replace us and possibly kill everything we care about But again you have this prisoner dilemma situation where while it benefits everyone to not do a dangerous thing individually you may get financial benefit from being the last one to stop So there is this game theoretic problem between different labs where they all want the government to step in make it illegal to develop super intelligence but they want to be the ones with the most advanced system at the time it happens So they all continue development until everyone else stops Really so you’d say they actually want to be kind of stopped from doing what they’re doing but they can’t draw out of the competition right so if they stop unilaterally they lose investors They lose their dominant share of the market So they would be replaced as a CEO for profit corporation They can’t do that They need an extern external force government regulator to come in and uh regulate them even if it’s just security theater uh otherwise uh they have to be at the cutting edge and cut corners in terms of safety to develop a more capable model and we see it in news reports I think there are whistleblowers now saying open AI has significantly cut their testing and safety budget So as a kind of ordinary Joe it’s quite hard to understand who these people are that would keep going with something that they know could endanger the entire entirety of humanity What kind of people are they oh very ambitious I mean if you’re talking about creating God and controlling future light cone of the universe in terms of economic value this is this is very ambitious people But uh it’s not surprising We have historical examples of people developing technologies which at the time they were not sure if they’re going to destroy the world or not Nuclear weapons is a great example So is there a kind of possibility of somehow stopping the development maybe governments could step in or something like that So it’s definitely better to add additional overhead maybe legal overhead The companies need to deal with legal hoops they have to jump through where part of their compute budget goes to lawyers That would slow down the process I don’t think long term it will actually completely stop this but uh one thing we need more of is time for sure How realistic is it stopping this i guess some other companies wouldn’t really like it I mean what obstacles would there be i I think it’s probably impossible to actually stop this if one jurisdiction does a good job with it They just move to a different jurisdiction It’s an international problem There are so many different countries with different regulations Even if the leaders of this uh technology US China had a complete moratorium in place other places would quickly take over So would it just drive things underground really or maybe to some countries that don’t comply so it really depends on the size of the compute effort If we’re talking about something as large as Manhattan project offered billions of dollars of servers and training it’s a lot easier to notice and monitor If a smaller model can be trained with a lot less resources and we saw examples of it coming out of China it becomes much harder to notice and regulate necessary processors compute available to those efforts I’ve also read about the AI centers a lot of them they need tremendous amounts of power tremendous amount of water to to run things So I guess that also kind of limits things somewhat right that’s another tool we can use to limit the largest projects Uh but uh this technology becomes more a lot more efficient very quickly So if today it takes a billion dollar center to train a model next year it may be a hundred million year after it’s 10 million and so on until everyone can do it on their laptop So you’ve devoted your career to AI You obviously don’t just want to see it completely wiped out What do you see as an acceptable level where do we draw the line to how to use it well the soonest we can stop is today The technology we have today is already more powerful than an average person in terms of their knowledge and capabilities It has not been actually deployed to the economy We can automate probably half of the jobs We can get billions trillions of dollars of free labor cognitive labor out of those systems And that has not been priced in It is not already something monetized So for the next 10 20 years we can just continue deploying existing models and see economy in a hyperrowth mode So kind of keep going with what we’ve already got We don’t need to take it any further basically deploy it So we saw it with other technologies For example video phones were invented in the 70s It wasn’t until iPhone came out that people started actively using FaceTime and things of that nature So it took 20 30 years to to get there And it’s the same with those models The model coming out today will not be in a factory will not be in a taxi for many years And until it is it is not fully integrated with economy It is not monetized I mean you’ve created a career in AI You’re kind of well known in the field I mean publicly saying it’s dangerous and that we need to stop it It’s probably not the the best career move I mean what what’s driving you to do that well I don’t really care about career moves I want to say what is actually true scientifically And at the time there was very few people who even looked at the problem Today we have Nobel Prize winners touring award winners thousands of computer scientists all agreeing with me So it is not a fringe position That is a majority position If you survey any large machine learning conference you get a consensus that yes this technology is going to get to human level Yes it is dangerous and pdoom levels at like 30% for AI researchers So is it enough do you feel there’s enough of a kind of surge of support for putting a cap on things to kind of save us there is enough talk and agreement There is not enough action Okay What kind of action would you like to see well for one nobody should actually be working on developing super intelligent systems You literally getting us all killed Now there’s several types of risk when it comes to AI One is X-risk which we’ve been talking about this existential risk There’s also S risk and Iris risk and can you tell us what S risk is and give us some examples of that right So some people are worried about even worst case scenarios such as uh suffering risks You’re not just uh dead but for some reason those systems took over and placed humanity in a position where we are so unhappy we would prefer to be dead So kind of digital hell equivalent uh either things you care about are no longer permissible for you to do either explicitly or implicitly All the art all the poetry all the games everything is no longer something humans can partake in because either there is competition for those jobs or there is just no economic support for it So right now a lot of people get meaning out of their careers you are a journalist Uh somebody could be a medical doctor But if all that is automated all those jobs are gone A lot of standard sources of meaning for humans disappear And so you may still have some unconditional financial support from taxing all the robot labor and AI but uh you don’t have any meaning to it So now we need to figure out what to do with all that free time for billions of people And a lot of people don’t do well without structure where they have some outside purpose to their existence I mean X- Risk and Srisk are not there But with this I risk we’re seeing already so much AI art It kind of floods your timeline on social media I mean you look at a photo or a video you don’t really know like is this real did it come from AI you kind of have a feeling of mistrust I mean are we already in the kind of high-risk zone in a certain way yes There are quite a few jobs basic design is one where you no longer can contribute AI will do a better job quicker faster and we’ll see it spread to everything else books movies all the creative outputs of people We also have this kind of transhumanism happening for example Neurolink and things like that you talked a bit about your your thoughts on that Right So right now while those systems are still tools they are not full agents They are not smarter than you Having a direct connection between your brain and a computer gives you an advantage You have access to internet You can communicate very quickly It’s an improvement But at some point that tool becomes better than you smarter than you It’s a full-blown agent And at this point you become a biological bottleneck What are you contributing to the system if it is more capable it’s not obvious that you have anything to contribute You just slow it down So either explicitly or implicitly you are removed from this hybrid system You also kind of opening up a back door to your brain for a system which could be malevolent which is again leading to possible asterisks Finally what gives you hope for the future well there is always a possibility that I’m wrong Maybe tomorrow someone will come out and publish a beautiful proof showing that yes you can control super intelligent systems indefinitely and here’s how we can do it and then it’s all glorious utopia for everyone It’s also possible there could be difficulties in creating systems which are that capable It doesn’t seem likely right now but maybe there is uh additional inventions we need to make and it will take longer to get to that point So there is some hope It’s also possible that for game theoretic reasons those systems will decide to play nice for a long time accumulating power trying to get us to respect them trust them even if later on they decide to perform a treacherous turn against humanity Professor Ramen Yamposki thanks for joining us Thank you so much for inviting me