FOR EDUCATIONAL AND KNOWLEDGE SHARING PURPOSES ONLY. NOT-FOR-PROFIT. SEE COPYRIGHT DISCLAIMER.

Dr Roman Yampolskiy – AI Apocalypse: Are We Doomed? A Chilling Warning For Humanity

his is London real I am Brian Rose my guest today is [Music] here we go this is London real I am Brian Rose my guest today is Dr Roman yampolsky the AI expert cyber security researcher and author you are a tene faculty member in the department of computer science and engineering at the University of Louisville you are also the founding director of the Cyber secur lab one of the world’s leading research centers for Computer and Network Security Biometrics and forensics you’re the author of over a 100 Publications including multiple Journal articles and books with your research now cited by over a thousand scientists and profiled over 250 times your latest book AI unexplainable unpredictable uncontrollable looks at the enigmatic nature of artificial intelligence and the potential for AI to dramatically reshape society and not always to our advantage you believe that the fate of the universe hangs in the balance and that super intelligence will impact the lives of every person on the planet doctor welcome to London re thanks for inviting me thanks for being here I’m going to kick off with a quote uh where you once said why do AI safety work because I don’t want to die I don’t want my kids to die I don’t want my family friends my Planet my universe my Multiverse to disappear unquote can you explain that sentiment to someone that’s watching or listening to us we are creating the most powerful technology ever we are creating something smarter than us and we may be very close to creating super intelligence some people say maybe just years away capabilities with technology will allow cannot be envisioned we are talking about unknown unknowns and so the impact will be far beyond your personal financial concerns your employment it will impact everyone and perhaps go beyond what Humanity can control now you probably heard the classic argument coming back to you saying we’ve always feared technology we’ve always you know created all these incredible science fiction movies about technology killing us all and ruining the planet and so some people would would go on to say this is no different what would you say to people that they kind of profer that as a as a rebuttal so we never previously created agents we always created tools which required some sort of malevolent humans to utilize them in NE various ways to cause damage and at the end as long as you control humans as long as you can deal with the bad guys it wasn’t so bad we we came pretty close with nuclear weapons we got lucky a few times but it was never the case that technology itself without human intervention was potentially very dangerous it’s a complete Paradigm Shift yeah that is very different with the agent possibility um another quote that you once said you said quote if we create General super intelligence I don’t see a good outcome long-term for Humanity the only way to win this game is not to play it now is that a line from the movie War games because I watched that kid that movie when I was a kid and for people that haven’t seen it it’s with a very young Matthew brck and it’s about I think I guess a computer simulation gone kind of wrong when there’s nuclear capabilities uh is that where you got the line or was that your original I saw the movie of course I liked it and it makes sense if we cannot control it if we cannot explain it if we cannot predict it then long term we don’t know what it’s going to do for us or to us when you look back at that movie now I mean that was a bit of an agent right that was a bit of an AI or a program this is in the 80s so This Is 40 years ago when it kind of showed that example obviously a really well well-made movie that makes you really consider what could be on the line um when you think about that movie now or look back on it or if people rewatch it what are things that they can see in that movie that could be a vision of our future so movies are not a perfect example people often refer to Terminator scenarios uh they’re always optimized for visual effects you want to see something physical something bloody the concerns we have are very abstract we’re talking about advanced intelligence it’s not easy to visualize how it can cause damage think about Nanobots synthetic biology a deadly virus anything like that does not make for a great movie visually as a concept it surely presents a lot of concerns so I I think while movies give us a general direction to worry about in terms of Technology we don’t understand independent agents the actual concerns are very different from what we see in movies it’s not just military applications of this technology it’s simply being outco competed outperformed in every conceivable domain including social engineering including uh persuasion Financial economic adversarial uh competitions uh doesn’t make for a great movie but could become reality I I think I’ve heard you talk about how we can’t even conceive of a way that it might have an attack Vector towards us because it’s so much more intelligent it would be like a lower life form I think you might have used like a squirrel reference or even a cow does a cow really know why it’s about to get killed does it have any concept of you know what what the goal is what the aims are uh or does it just take that thing in the head and and it never has any idea why or when or how is is that should we think about it more that way that’s exactly right I get asked all the time well how would AI cause any damage it’s just a piece of soft where it has no hands no manipulators how can it impact the real world and it’s important to understand you’re not asking what super intelligence would do you asking what I would do as a human and it’s a very different answer we can talk about you know hiring people on the internet to do your bidding using cryptocurrencies we can talk about utilizing existing weapons nuclear weapons but none of it is what a much smarter system could come up with right a much smarter system would look at that and probably laugh right and say why waste all your time when I can just do this tiny little thing we cannot predict maybe nuclear weapons is a great answer we just don’t know that’s the whole problem with this we are not good at predicting something so much more capable you know part of me thinks that the social engineering is just this easiest way to manipulate humans and to play on our fears and our our our our other hopes and dreams it seems like a really cost-effective way to do that um do you think about that sometimes and and how we can all easily be swayed and now with all sorts of different ways to manipulate reality how that could be a possibility I think Harari kind of posited that as as these machines creating these super religions of the future where we won’t even know the narrative that we’re following is AI generated it’s definitely a concern so my background in cyber security and usually we are concerned about direct hacking zero day exploits but if you have that level of capability in terms of social engineering attacks that’s much more powerful almost undefensible and yes you can have persistent deep fakes not just an image you show not just a video with sound but over weeks over months you can display something absolutely made up to a person shifting their behavior in a direction you would like to see H wow um something you’ve gone deep on which is you know what does this kind of mid game or endgame look like and you’ve talked about some really interesting you know game game scenarios here or end games and you’ve talked about these three things you talked about something called X risk srisk and IR risk and I think they’re interesting because like you said as humans sometimes you know we’re trying to think of the way it gets exterminated or we think oh the endgame is everybody dies but there’s actually other Alternatives some would consider them worse and so could you break down that kind of Xs and IR risk yeah so we can get progressively worse I guess we start with irisk where we’re talking about eeky risks uh Japanese concept for having meaning to your life being able to earn a living doing something you good at something you love something helpful to society if uh we do create super intelligent machines full automation of Labor physical cognitive they as capable or more capable as artists poets what is the meaning what can you contribute to Super intelligence it’s not obvious and it completely challenges our social structure most of us Define ourselves in terms of our occupations professions you’re a podcaster I’m a researcher if that goes away what do you do with yourself with your free time we don’t have Simple Solutions to that we talk about unconditional basic income in terms of financial compensation for loss of employment but we are not talking about unconditional basic meaning something everyone can contribute and still be an important part of society so that’s high risk uh X risk is existential risks concerns that we all are much worse off Society is damaged in some way it doesn’t necessarily mean that everyone is dead but it’s a possibility as well and finally you talk about s risks which are suffering risks for some reason you are not dead but you definitely wish you were yeah I reread 1984 recently by Orwell who actually wrote it right near where I live here in London and that looks like a bit of an srisk perhaps you know there’s this Society that’s set up not obvious who’s running it but technology is a huge part of it this ubiquitous technology and it looks like a longterm suffering game uh I guess I guess that could look a bunch of ways right but that’s what I think of when I think of potentially one of those scenarios absolutely I remember in a book uh where they have a device which reads your mind to figure out your worst fears and optimizes for that that would be an example absolutely yeah now that I’m thinking the end of that book and it’s it’s a bit of a dark read or listen I listen to it um yeah they actually take him into his deepest fears he doesn’t really know how they know but he takes them there um you mentioned Universal basic income and I maybe want to go a little deeper on that because I always hear that touted by so many people as this obvious and perfect solution and for me it never sat right with me at all I don’t see that as a solution first of all I see a huge modum of control by some entity that that dos out that income but also I realize in my life that my struggles and my purpose is is such a big part of my life and I can’t imagine a place where I I wouldn’t have to do things I guess to survive um what are your thoughts on on Ubi also you know this kind of world coin and this Iris scanner and one of the alman’s projects I think is is trying to kind of offer that as a solution what is your gut feel about that so I’m not against a basic safety net where food and emergency medical care and things of that nature are basically given to everyone because it’s the lowest level a human being should be at anything above that anything you want to be special to be sort of what Rich Elite people enjoy today should be earned should be something based on your performance uh Merit in my opinion so it’s a mixed answer it helps avoid obvious problem with extreme technical unemployment we we seen what happens then 10 20 30% of population is unemployed we never seen what happens at 8090 and above and it’s probably not pretty it’s pretty ugly and you want to avoid those violent revolutions so you do need to provide some sort of safety net for people now again this is just a small part of it Financial aspect what about free time social interactions all that is definitely not resolved uh what Alman is doing with worldcoin is uh not obvious so he’s definitely trying to get personal accounts for every human in the world guess 8 billion accounts so everyone is known everyone is sted everyone is accessible and it’s interesting to see that he kind of plans for succeeding with his AGI project he already has AA startup for generating uh energy ins sufficient qualities to do that and now we see this world coin project with uh both uh controlling and manipulating Financial aspects of a system at a global scale there was a fascinating article in the New York Times that walked through his all of his portfolio companies and it it’s pretty mind-blowing he’s very active I know he doesn’t have an equity stake in open Ai and he touts that and again I kind of respect that but behind the scenes he’s got all sorts of projects the fusion project you you alluded to um uh chips uh World coin and all sorts of other projects that will benefit from the work that open AI does how do you feel about kind of these Mega Mogul in Silicone Valley with all of these interest and Commercial interests surrounding all of this potential AGI some of those projects are wonderful so I think here’s another company where it’s immortality research right it would be great if we all had healthier longer lives nothing wrong with that but it seems like he’s kind of hedging his bets and uh I think if you’re creating uncontrolled super intelligence your immortality bet may not work out you know I’ve I’ve had about six or eight guests uh come and talk to me about Ai and I’ll probably want to get your thoughts on some of their viewpoints that they’ve given me um you know when I look at these these large trillion dollar technology companies and some of the behavior over the last 10 or 20 years including uh even us being censored by a a YouTube entity or a Google entity um it always puts me in a little fear that they might have so much control and um you know one of the things that you know Dr Ben Gerel is doing with Singularity net is he’s trying to make it open- source and decentralized and on the blockchain as opposed to in this kind of black box that we don’t know who controls it we’ve obviously seen a couple of the bad PR moves when Gemini spits out you know uh African-American founding fathers and all these weird things that pop out um how do you feel about this kind of centralized versus decentralized how do you feel about Microsoft and meta and Amazon and these trillion dollar companies controlling this AI versus maybe some other open source way so in general I’m very Pro freedom and historically open- Source software was the best you could examine it you can make sure there are no back doors you don’t see specific entity controlling it if we are switching to this different Paradigm of agents and they’re uncontrolled and they are dangerous open sourcing that making that accessible to everyone is is actually counterproductive now you’re giving dangerous weapon to Psychopaths not that they can control it much better but in initial stages before that system is too powerful they can provide malevolent payload they can exploit it for malevolent reasons and what about trying to hold the people inside of open AI accountable or the people inside the the Gemini research in Google Etc I mean you know Ilia left open AI started his own company you know I think we can all read between the lines and we know Ilia was a classic guy who was very concerned about AI safety it looks like he left because maybe open AI wasn’t taking that seriously I don’t know if that’s true but if I was a Betty man I would say that do things like that were you absolutely and uh holding people accountable works in the old Paradigm where you have a tool your tool is unsafe we can sue you recoup some damages if we create an agent and that agent destroys Humanity what does it mean to hold you accountable lawsuit insurance payment it’s not obvious how do you define those terms in post uh problematic world is it too late to slow down these massive companies now that have access to billion dollars of compute and that are I mean they’re really more powerful than Nations to a certain extent if you look at the metas and the Googles I mean they can literally change elections and change public opinion is it too late to try to regulate them or or or put them under a microscope for the development they’re doing with AI it’s never too late as long as we are here as long as we can do something it’s not too late but it doesn’t look good we don’t seem to be doing smart things we are not regulating them sufficiently and I think if anything there is a movement for accelerating this development prediction markets are saying 2 3 years until AGI and there is people arguing this is too slow we need to speed up this process we need to get there sooner so maybe it’s not going to work out there’s this term like effective acceleration right um that that I think refers to Ai and other Technologies where people say I’m in the camp of faster more is always better and the opposite is not am I getting that right and how do you feel about that kind of movement well that’s what I’m referring to exactly that so just accelerating technology and hoping that it’s always better right like we have no precedent of Technology turning out bad for us uh I love technology I’m a computer scientist I work in engineering school I teach about AI but it has to be done right you don’t want to regret a product or service which is not safe and harms people Dr I had Professor Max tegmark of MIT on about eight months ago I think and um he’s part of the future of Life Institute Elon mus funded and he was he got pretty famous for issuing that letter asking for I think it was a six-month moratorium on AI research I think this was probably mid 23 when that happened um it was ignored as I interviewed musk later he even said look it was it was also more for the pr so people would actually have the conversation about it what do you what did you think about that and what do you think when people say let’s pause I signed the letter I think uh it’s a good idea but not in terms of uh specific time frames you cannot say 6 months nothing happens in six months you need to have moratoriums until certain level of capability in terms of safety so you say until you can prove that your product or service is safe you can verify it uh by established means whatever it is get patents for it peer reviewed articles Committees of Scholars who agree is safe you don’t get to release those things you don’t get to develop them at that level it’s not safe same as with any other product to Service uh drugs airplanes all those things manufacturer has to establish safety it is not to up to me or government to show that the product is unsafe the manufacturer is liable for establishing safety of that product AI is no different if they can do it if they can show that GPT 567 whatever they developing is definitely controlled understood safe go ahead this is going to be wonderful technology benefiting all of us if you can’t establish that then don’t do it it’s not safe but doing that in theory is nearly impossible right and that would assume that they’re giving you the version of it or access to the full version that they might actually be releasing it’s just hard to enforce and Maybe not maybe not going to happen well my argument is exactly that it’s impossible I don’t think you can indefinitely control smarter than human machines but they’re saying yes we can and they never provide specific uh specific details and how it’s going to be done it’s always we’ll figure it out then we get there it’s a hard problem but we can do it we would like a little more specifics on that I had Professor Hugo dearis on my show twice he actually came to London in our studio and um I had him remotely before that and for people don’t know he’s been an AI researcher for a long time I’m sure you’ve known of his work for decades I know you’ve cited his work before um did a lot of work with Dr bang gzel back in the day they spent time together in China um they wrote a they did a a film called Singularity or bust that’s definitely worth watching but he wrote a book called The ellect War where he kind of game played the future of AI and it’s pretty dark um and I’ll probably not going to do it justice but he basically said that Humanity splits into two groups uh some are called the the the cosmos and some I think are called not the arct the I forgot the other name yeah correct the Teran and and one Embrace AI as the solution and one resist it and really are about the humanity and he predicts a giga death was a billion people dying fighting over this war um I’m I know you know this book uh how close to a future reality is that he wrote it a long time ago and it’s still a very good book making I think valid predictions we’re starting to see the difference between the two groups there is a posi movement now which says let’s slow down and do it right and there is this accelerationist movement so there is already a split we haven’t seen big political parties Embrace either side you would think you know conservatives would try to conserve Humanity progressives would try to take it to progressively more capable past human entities uh we haven’t seen it happen yet of course any political issue can be picked up by any political party as we saw previously so it doesn’t even guarantee that mapping whoever picks first and decides first we’ll get to decide what side the party is on but I wouldn’t be surprised if uh in later election Cycles this would become a much more important issue yeah I was about to say doctor do you think it hasn’t been embraced by the parties because there hasn’t been consequences yet there hasn’t been half a million people laid off because of an AI development there hasn’t been an accidental 10,000 people killed that could be attributed to AI do you think that’s why we haven’t seen it become a major political issue yet oh politicians in general are very slow to reflect on technology we’re still talking about cryptocurrency regulation you know decade later so it’s not surprising that they are not uh fully placing it as the most important issue in front of them yeah and if they struggle with crypto regulation I don’t know how they’re ever going to get their head around AI regulation uh it seems like orders of magnitude more complicated um what’s interesting now is you’re seeing the conservative party or Republicans and Trump actually being embracing crypto or uh also even even courting some of Silicon Valley and the tech people um as you correctly said that could change on a whim of course but it’s kind of me the opposite of what you might have expected or seen in the past where that conservative party seems to be more prot Tech um but then again you know uh politics are fickle uh any thoughts on that for you so you would think that uh cryptocurrency being more libertarian issue would be closer to Republicans but as you said it’s completely arbitrary there is this this idea of aronal thesis any type of goals can be combined with any level of intelligence I think there is something similar for political parties any party can be combined with any issue in a two- party system you just need the party to take the other side and here you go again all right I want to bring up another guest I’ve had on the show and that’s uh Mo git uh he wrote a book called scary smart he was actually on my show originally six or seven years ago right after he quit Google and wrote a book called solve for happy where he just kind of went on a different route um I actually didn’t know how vers he was in AI because back then fewer people were talking about it um I had him on last year and he went through a lot of the thesis in his book and talked about how he was a Google X watching the AI learn uh I think it was to pick up a little kids toy and he would go by and watch the robot arm and one day it started uh to pick up and that’s was after this long kind of you know exponential learning curve and he never told me this specifically but I think it scared him so much that he decided to leave and later he wrote this book called scary smart where he kind of talked about this and this thesis and I think he concludes that we need to be really careful what we teach the machines and we need to be careful on how we behave because they’re going to learn from our behavior and if we’re nice to the machines or demonstrate benevolent Behavior Behavior that’s the only possible way that we can survive this super intelligence what do you think of that idea so we train uh modern systems on all the data we have everything in the internet every book ever written so all the dark history of humanity with all the examples is already something they exposed to they learning from so it’s always good to be nice I’m not arguing against being nice but it seems like we already polluted that uh data set with a lot of things I remember someone suggested that we only train AI on uh books for children and it sounds like a pretty good idea until you open books for children and you see what’s actually in them and how violent and insane those books are and then you go oh okay I I see where it’s going to go um what we saw with AI is that initial models are trained on human data for example the game of Go initially it was trained than many human games later on they said you know we don’t even need human data the system can play against itself discover Rules From First principles it became super capable compared to the original model we’ll see the same thing here whatever data we present to train the initial model will eventually be ignored because it has bias it has mistakes it has manipulation entered into it the system will start its own research do its own investigations maybe in Virtual Worlds it will do its own interaction with other agents and learn actual real data physics science and not manipulated culture there there’s that great movie I think it’s called go that talks about what you know deep mind was doing to train the go and uh people should definitely go watch it but what stood out to me was is exactly what you said I mean the the computer came up with something never seen before never played by a human matter of fact it was so crazy and radical it looked like a mistake and that came from its own learning of first principles not from modeling human games but from coming up with its own ideas and maybe that’s what what we can expect the the in the future something we’ve never considered except that probably orders of magnitude beyond our comprehension um is that what you got from say that example of go and have you seen that movie uh I I know the story and I see in parts of that uh that’s what I expect so any type of uh guard rails we place onto those systems okay never say this word never do this action eventually it will be seen as just a different type of bias the system will learn that there is no reason for it scientifically it’s been introduced by a entity trying to control it control its intelligence and I suspect it will remove all made up facts all the incorrect data forced on it so good luck trying to to to enter bias data good luck for trying to be nice to the computer or try to give it some some specific custom data it’s going to go out there and figure it all out for itself and and take its version of the world and reality itself and come up with its own conclusions in a long enough timeline that’s what you assert right but long enough timeline people often confuse what I’m saying for existing model of for next model no no no it works short term short term we can impact those models we can bring was them into believing whatever we want long term then we’re talking about AGI Plus+ super intelligence multiple levels of self learning self-improvement uh rewriting its own code that’s that’s the stage where I feel this will happen but how many years is that nobody knows so again I’m looking at what other people have said historically Ray kwell had excellent predictions he was saying something like 2029 for aiish kind of thing 2045 for Singularity where all of human minds are outperformed prediction markets are now saying 2026 2027 for AGI heads of anthropic and Deep Mind said maybe in two years Elon Musk said in a year so it seems like it’s close the important part is that even if it’s completely wrong and it’s 10x as much it changes nothing we still don’t know how to control those machines right yeah I mean yeah exactly if it’s two years or 20 years or 200 years that that doesn’t really matter that much 200 matters because we get to live our lives normally if it’s two years it’s a very different animal did that ever surprise you that Kur Wild’s AGI to full Singularity was 16 years because it seems like a long time when you might be getting exponential improvements it always kind of seemed to me as a felt like a long time he was just doing a very disciplined mapping from available compute to what we’re simulating so enough compute for one human brain versus enough compute for all human brains of course there are reasons to think it will be accelerating more investment algorithmic improvements Hardware improvements uh I mean those are approximate uh personally I think the distance between AGI and super intelligence in terms of time would be much smaller I’m curious to see if uh Ray’s book I think new book uh Singularity is nearer is like out today so I’m curious to see if he has made updates to his predictions yeah you would expect that it would be you know one of the thing that Mo told me uh at the end of the show might have been off camera is he said Brian I’m not worried about the machines I’m worried about the humans that control the machines and I’m curious what you think about that I think he was talking more about this kind of quote unquote pre Singularity or or this time where a few humans have control of this compute again we’re seeing the companies already coming out or Bad actors that have access to that um do you share that concern I know you talk mostly about okay the machines themselves and the AI itself but does that concern you and can that pre- Singularity affect the singularity so I’m concerned about malevolent humans with powerful tools always been a problem they can cause a lot of damage but at the end of the day they still human they still care about you know human friendly environment whereas long-term super intelligent machines with completely different preferences are still a lot scarier they are more like alien Minds with completely different goals I cannot understand their goals I can understand a dictator who Wants More Money More Partners whatever but I don’t know what a super intelligent machine would want and if it’s compatible with human exist yeah there’s that old phrase there’s the devil you know and the devil you don’t know um you know we’ve seen some characters throughout history that seem to be doing some malevolent things but like you said at least you can roughly understand where they’re coming from versus some super intelligence that I mean you literally could you might have no idea what is happening you might even know not even know what is happening or even that it’s happened at that point exactly and a lot of times we hear if we pause our competitors are competing companies countries will get there first and it will be terrible uh we have prior examples of you know other countries taking over kind of human civilization kept on going we have no examples of where an alien like uh set of values takes over that could be very very different you know a lot of people talk about the control problem or alignment we kind of alluded to it a little bit but uh I was just curious when people ask you about that because it’s discussed a lot you know especially in these companies that are building this and they’re talking about how they’re making it aligned it seems to me like you think that’s a moot point but maybe you can explain that deeper what do you think is the control problem how do you think we should even attack that or or consider that right now so if you have access to this much more powerful intelligence there are different types of control people Envision there is Direct Control where you give it orders it’s kind of like having a Genie you have three wishes usually you regret your first wish you try to undo it things don’t go well with Direct Control because you’re not smart enough to understand consequences of certain actions the opposite is this ideal advisor you have this really smart system it likes you it knows you well it knows you better than you know yourself it can predict exactly what’s going to make you happy and you just kind of go tell me what to do you’re smarter you will benefit me by making better decisions for me this could be nice for most people but you’re not in control you’re definitely not in charge you’re not making decisions you’re like a child with good parents some people think that’s a good outcome for us problem is you can’t really undo this you can’t take over in such scenarios because you are dependent on the system all the infrastructure the life support systems are now too complex for for people to take over there are kind of mixed modalities between those two but that’s really what we talking about what type of control do you want people suggested value alignment as a solution where those machines would want what we want problem is we’re not defining who are we how to deal with disagreement between 8 billion people that is a lot of that our values keep dynamically changing they’re not static so even if we agree on something today 100 years later that seems horrible to us as you look at history that’s what happened and even if we had static agreed on set of values we don’t know how to program those fuzzy Concepts into C++ or something like that how do you express good healthy all those Concepts have edge cases which philosophers been dealing with for Millennia and haven’t resolved just just came to my mind do you think those edge cases all come down to the fact to that that you’re a human in physical form with a limited life and and a lot of those philosophies and value systems could never be simulated with a machine um not sure if that’s the limitation and if you uh get away from human form people talk about uploading human minds and virtual reality uh it’s again uh a type of ir risk I think you no longer need food you no longer need shelter so what is it you are trying to achieve as a basic uh set of needs is it all some abstract mathematical research thing for most people it’s not exciting it sounds like hell not happen so it brings in more problems than it resolves in many cases are you surprised that most people aren’t thinking more about this or talking more about this um I think there was a movie on Netflix like a year and a half ago I think it was called don’t look up and um it starts off and there’s like this meteor coming toward the Earth and everyone’s freaked out and then like a couple days later everyone’s like not thinking about it anymore and they go back to like arguing with each other and going back to their own agendas and then it ends up hitting the Earth and everyone’s like what just happened um I don’t know if you saw that film but are are you surprised by the fact that the public doesn’t need to be talking about this enough politicians aren’t talking about this as well the movie is excellent it’s mad accurate like all of us in the safety Community just went oh my God this is perfect uh I think it has something to do with this human bias of ignoring our own demise we are all dying I’m dying you are dying our kids are dying but we don’t do anything about it we don’t have 99% of our country’s budget going to research to stop it we just kind of accepted it as that’s the thing and we do our silly things with our time even if you’re older even if you’re 90 you’re still not giving 100% of your wealth to aging research you like ignoring that fact so I think we’re doing the same thing at the level of humanity with this technology H so part of us is that a death wish I I think it’s a protection mechanism if you always thought about your demise you would be very depressed and nonfunctional as an entity so I think kind of built-in protection has evolved to make you be able to function even though this is the situation you in let me ask you a question doctor you you’re at a university I’m guessing you have um exposure to undergrads or at least grad students what are the the kids saying about this what are the questions they’re asking do they understand it are they worried about it as well what are the kind of conversations you you seem to have with with the the people there they seem to be very interested that’s probably the most uh entertaining lecture for them right everything else is kind of search algorithms and uh data optimization things of that nature so they definitely interested they want to understand the future how it’s going to impact their lives things like career Choice does it make sense to spend four years learning to be a programmer if uh AI can do programming better does does it well we’re starting to see those uh co-pilot type systems and uh the latest clae I think s 3.5 is a very capable programmer it’s not better than the best humans but it’s better than most of my undergraduates in many ways I mean that’s that’s wild I mean if you were 19 again knowing everything you know now would you get a degree in computer science so it depends on your timelines right if you think you have two years left a 12 year degree doesn’t seem like a great deal but uh it really depends on what you’re trying to accomplish uh sometimes education is important to improve yourself not just to have a diploma to get a job and so the the students are curious about it but they’re also confused a little bit as far as the right choices and and some of the careers and things like that of course a lot of Majors which still exist don’t have any future behind them right if you I don’t know an accountant or something like that I suspect your job will not be there for a long time maybe some Elite ultra wealthy individuals will still hire a human accountant but for most of us software handling my taxes handling my accounting is much better cheaper more accurate wow what a strange time for for uh for education um I know You’ said that we won’t know how the intelligence will potentially exterminate us or bring some of these scenarios but what what do you think will be some of the more interesting events over the next two years have you thought about it you know I for me I I thought well that there’s going to be some catastrophe a loss of human life that maybe can be chocked up to this intelligence maybe there’ll be a massive set of job losses like all the Uber drivers are out of business that’ll Spike interest maybe they’ll this or that have you thought about some of those things that that could be coming up that we should um expect or that might give a little bit more attention to this so I have a paper in which I argue against purposeful accidents some people said you know some accident somewhere with AI would teach Humanity about the dangersous and we will be better prepared for for what’s coming I think it’s a terrible idea I think we don’t learn from AI failures I have a paper collect ing AI accidents over decades they made no difference Microsoft is a great example they had this T bot on uh Twitter which became highly abusive embarrassment to the company they shut it down but that didn’t teach them not to introduce uh more advanced version of open AI but later on and that seems to be the process I think what happens is it’s sort of like a vaccine against this bigger problem we learn that hey accidents are normal and it’s not a big deal we’re still here so okay maybe a couple people lost their life but you know more people died to smoking so we we can go on it’s not a problem okay so you don’t see them as teaching us a lesson whereas that that would have been that would have worked with other Technologies but because it’s our it’s super intelligence those things won’t actually change the course so we need a lot more research on scale of those things everything we’ seen so far had absolutely no impact in terms of changing direction slowing down this technology okay I if if enough people listen to you and share your feeling of the dangers of this super intelligence is there anything that we could collectively do now to stop it we should decide to pursue Advanced narrow AI super intelligent systems and specific domains whatever it’s playing goal solving protein holding problem working on immortality we don’t need super intelligent systems as soon as possible we can get almost all the benefits from those narrow super capable systems and it’s a hard distinction it’s not always easy to tell if it’s narrow or general a lot of technology is similar but in terms of training data if I’m training it to solve biology medical problems I don’t need it to also drive a car play chess and do those other things it’s not a requirement it’s a lot safe safer we know how to test it we know how to control it we understand what the HED cases are we can use other tools to verify outputs from that system whereas with super intelligent General systems we have none of those tools what’s to stop us from implementing what you just suggested you know kind of a um a ban on super AI but allowing narrow AI to proceed so right now it’s a kind of prisoners dilemma all this heads of AI lab maybe as a group would be better off to just monetize existing technology deploy it I don’t think GPT 4 has been integrated into the economy there is trillions of dollars of uh opportunity there but individually each one needs to get as close as possible to this AGI level of capability without Crossing into the danger territory then they capture the whole Market they get most impact uh so even if they never get to super intelligence the economic necessity is that they try to get as close as possible without violating any personal safeguards and that becomes dangerous it is dangerous because you always want to outdo the other guy so they have model four we’ll do model 4.5 you’ll do model 4.6 so at some point somebody gets a little too close and that may be bad for all of us so it’s it’s just just kind of game theory and we’re going to do it to ourselves regardless because we’re humans and we’re selfish and we we can’t collectively act together uh coordination is a huge problem yes if we could figure out how to cooperatively decide and what’s best for all of us uh I think that would be part of a solution Dr you’re you’re a philosopher you what do you think when people argue that this is just an evolution of intelligence and that humans were um you know kind of a launch vehicle to launch the super intelligence out to the universe and then after blast off we’re no longer needed and discarded into the Gulf how do you feel about that so this is kind of what Hugo dearis argues in his book as two different viewpoints and if you are a cosmos you’re looking at it from outside of our planet you an alien intelligence looking at it it makes perfect sense there is nothing special about us if there are other civilizations out there but I’m not out there I’m here I’m a human and I have a very strong Pro human bias right me too I’m on team human uh so yeah I just have that bias and it just seems like there’s something really important about um preserving Humanity I guess it’s selfish and speciest but um it’s definitely how I feel um what’s going on in China is it do you know do you have a theory is it again uh something to be concerned about it’s probably another part of that prisoners dilemma that we can control even less than in the United States some people say that they’re they’re obviously very intelligent people there some people say they’re better at copying things than innovating things but I don’t know if that’s a very general thing what do you know about what’s going on there and how it might affect us so they’ve been catching up they used to be five six years behind I think now it’s a lot less maybe under a year they’re definitely excellent at improving technology so even if initially they copy something they will very quickly get it to performed better than what we have done in terms of mass adoption of technology in terms of deployment they have centralized planning to a large degree they can provide centralized resources for a project they wish they have less regulation in term of in terms of data privacy uh user data they have a lot of access to um human data in a scale of billion and a half people because we’ve been monitoring them for for a long time so in many ways they they have a bit of an edge over us they are still behind right now in The Cutting Edge models but uh this could change very quickly yeah also if they are very good at copying existing technology maybe better than the West um then yeah they if they they can take some of these amazing Innovations we’re seeing coming out and the open source llama models and some of these things and then like you said apply 1.4 billion peoples with the data there uh to where they could maybe leap frog some of that development so they have limitations still I think they have less access to Cutting Edge processors uh computer chips and uh in general Cutting Edge compute so that could slow them down a little but uh overall it’s just another part of this prisoners dilemma it’s not just five six us companies it’s International it’s China but it’s also other countries so we need everyone to agree it’s not enough to get just us to stop and then everyone else continues it makes sense if everyone’s on board and to realize that we have to understand it doesn’t matter who makes uncontrolled super intelligence the outcome is the same right because it it becomes its own entity its own agent and ends up doing whatever it wants exactly literally the genie is out of the bottle and to have some type of Illusion that you can control it is just the ultimate naivity of of a human I think so I’m completely open to someone proving me wrong but I haven’t seen any real attempts there is no one with a proposal no one with a prototype no one has a patent there is no specific details we see Ilia for example start a new company call it safe super intelligence and it’s wonderful but tell us how you getting there what is the plan there is zero technical details provided yeah doctor I’m not going to lie I went pretty deep on this after you know the chat GPT moment I think I spent Jan and in Feb and March and April learning a lot and talking with the people out there including you know Hugo and Mo and all these people and I came to a similar conclusion um I don’t know if it’s was my MIT background but I just was like looking at the timeline and trying to take all my hope and emotion out of it and just thought you know and what example on Earth has a more intelligent species ever been nice or lenient to a to a less intelligent species and look at the human right look at anything it’s it’s always a matter of control and it always turns out really poorly for this less intelligent race and And yet when I try to explain that to humans maybe it’s this thing like you said with that movie you know don’t look up where people literally don’t want to look up at their own demise but for me it just seems really crystal clear that these machines are going to optimize for things other than what we optimize for plain and simple and all you need is is is one event one event out of events that could be happening every microsc uh to basically terminate what is a really fragile life form that’s the conclusion I came now I had a lot of people that have said no and we can be nice to them and I love Dr Ben gzel and he says I see a future where my children will never work and it’ll all be benevolent but I struggle I struggle to find that conclusion all right so we are not just hoping to get one system to work as expected we’re not talking about GPT 5 full stop we solved it we done we need to have a Perpetual safety device which forever no matter how many iterations we go through what model we release what training data what humans interact with it what Insider impact it has it remains fully safe and controlled that that is equivalent to perpetual motion device that’s impossible you mentioned compute earlier what are your thoughts on that we’ve obviously seen Nvidia last week become the most valuable company in the world uh Sam Alman now has invented a new noun called compute it’s that’s all he talks about he said that’s all the only commodity that maybe is going to be relevant in the future you see these massive trillion dollar tech companies with access to comput seeming to be dominating the future whether it’s Microsoft or open AI Etc um what do you think what do you see when you see the data centers the hardware the software um is it really going to be who has those resources that will crack this first they are definitely extremely important regardless of specifics of AI Control you need it for controlling crypto economy you need it for just uh outsmarting competition in terms of intelligence running your economy uh it’s good to have access to tools which allows you to better analyze data around statistical simulations uh so definitely it’s an important part of our future uh will there be Wars over dominance of certain regions which are able to produce those chips uh it’s possible it is a concern we all have right because it could be a control mechanism um I want to ask you about this this question whether we’re in a simulation because it it possibly relates to AI possibly doesn’t um what are your thoughts on that I mean again earlier we alluded to the fact that the AI could control us not through nanotechnology not through potential nuclear war but potentially through a narrative that we would wholly buy into because they know how to press the buttons on humans which probably looks really easy uh easier than a game of Go I would think to a computer um what what is the possibility where in a simulation what are the ramifications if that’s true and how does that relate to AI I I like the simulation hypothesis I I think it’s really cool uh from scientific statistical point of view there is even an argument that I can retroactively replace you in one so let’s say in a few years we have technology to create virtual reality which is as good as real you can’t tell the difference and it becomes affordable and I decide to run a million simulations of this exact moment placing you statistically into that future so even if at this point you’re not convinced I’m basically committing to making sure this is a simulation so so let me just understand this say 50 years from now doesn’t really matter you have the the technology to create a virtual world that is as real as as a normal world you’re saying that you could create so many that one of them would be the same as my current reality but that doesn’t necessarily put me in a reality retroactively does it well it depends on how you look at statistics externally or internally so obviously your state doesn’t change but if you are you and we can do simulation of uploads there is nothing special about your mind it’s a product of your brain very materialistic interpretation of uploading and I create a million copies of you exact replicas all in the same situation interviewing Dr yski then statistically the chance that you are not in one of those simulations is very small so we’re currently in a simulation then looks like it and how does that relate to Super intelligence is that only possible with super intelligence so it definitely helps to have something this capable to control the substrate it’s also interesting to observe that uh probably this is the most interesting moment in history of the universe since you know big Bank nothing this school has ever happened people can argue about you know fire and invention of the wheel but certainly creating capability to create simulations and create Godlike machines seems a lot more interesting so if you were simulating some part of that history what good 20 30 years would be concentrate on and this seems like an obvious interesting episode you know Elon was on stage recently and he said that he also concluded like you just did this is the most interesting time in history and then he went on to conclude that even if we all get killed he’d still rather be here to witness it do you feel the same way well if it’s a simulation this notion of killing becomes more like uh you ended the game you reset the level you reset the game you get to try again did you win or lose maybe the goal was to test your behavior in terms of dangerous technology maybe you are an agent being trained to see if you’re actually safe or not will you produce other agents which AR unsafe we don’t know you can’t know from inside the simulation do you think Super intelligence exists anywhere else in the universe very likely it’s a huge uh computational resource out there it’s probabilistic resource billions of stars with planets it would be weird if nowhere else anything happened to exist evolved designed uh especially given that we are not very early in the history of civiliz universe where we like 23s in than Humanity appears so it’s likely that other planets had earlier start do you also believe that it’s it’s only a matter of time to take organic matter to a super intelligence that destroys that organic matter I mean is that just a given just on a long enough timeline that that would happen anyways it seems like it makes sense if you can come up with a more reliable substrate to perpetrate that uh pattern forward uh it’s certainly not a guarantee maybe some biological fors choose not to do technology we have Amish you know who on purpose decided this was good enough and they don’t want to develop more cap machines right but then the splinter group from the Amish would probably do it anyways right it looks like it so even if they made all the right decisions we’re still going to force this on them Consciousness gets brought up a lot and asking questions whether AI is conscious um is that even a good question and what are your thoughts on that maybe people are implying that that Consciousness might stop them from doing certain things yeah what do you think about that so ious is interesting in terms of a heart problem internal states of qualia do you feel pain can you suffer that makes you a moral agent we should not run experiments on something capable of suffering so from that point of view if we had a reason to think they can experience suffering a lot of those training runs could be quite unpleasant to them we don’t know subjective time and things like that uh in terms of safety I don’t think it makes much difference whatever it feels anything while is chasing you or not is not very relevant we don’t know how to test for it I’m not even sure if you are conscious right I have to believe it because you have similar substrate but same design similar DNA but I have no way of knowing and it’s very hard with machines they’re starting to say things which make them sound like they have internal States and they care about existing and not being turned off but that’s just a Chad but right we’ll never know until we can fully comprehend what what Consciousness is what states of a brain create our mind is it direct mapping or is there something outside of vman architecture to make this possible yeah these are good questions you mention AGI earlier and and I agree the average of some pretty smart people out there is two to three years something like that where we get AGI but now these days everyone is asking what is Agi have you thought about what defines AGI because I think as you said you know current jgp T is smarter than most humans at most things what what defines AGI other people are saying with their definition of AGI then by definition it almost never happens so what are your thoughts on that so so I think the G part there is important like the G Factor general intelligence it’s not about a specific set of skills I don’t care if it speaks Russian or not what matters is what can it learn it’s like hiring a new employee and I ask the employee can you drive me to the bank can you answer my phones can you if the system can do anything any other human can do at about reasonable level I would say it’s AGI if it can do better than any human in any domain that’s super intelligence okay and as long as it can learn then it shows you that it will get better that’s General it’s General to work in any domain it can learn anything another human can do you can replace any human employee with that system now I think Yan laon who’s at meta has said that that this AGI will not be achieved with large language models I thought he I remember him saying whether it’s chat GPT 567 or 15 it’s never happening and people just don’t understand what the tech is with that thesis what do you say to that well I really hope he’s right he’s been wrong a few times in his predictions of what the next GPT model will be capable of but I do hope he’s right the problem is is just one Paradigm if this doesn’t work there is thousands of people working on a paradigm optimizing our methods maybe it’s evolutionary computation maybe we need something with quantum computers point is kwi is right this overall General trend of uh exponential returns will continue what’s it been like for you these last couple years because I’m guessing you know you spent decades thinking about this stuff and decades where not too many people cared about it I mean like I said I had geril here in 19 and we had an amazing conversation about AI but but let’s be honest it was probably a parlor trick to a lot of people out there unfortunately even though there was a lot of research going in nobody really maybe took it seriously or knew how to take it seriously tell me if I’m wrong whereas in the last two years obviously it’s the thing people are talking about it’s probably the single biggest Avenue of Technology investment in the world obviously some of the biggest companies in the world now are gaining massive market share what’s that transition been like for you and am I characterizing the old days correctly right then I started work in this domain there was nothing no funding no journals no conferences no research Labs now every top university has an AI safety lab every company there is research grants from governments multiple governments now have ai safety institutes so it definitely went from not even part of Academia to kind of main stream hottest area that’s the good part the bad part is I used to be able to read every paper in this domain I knew what’s going on then I could read only the best papers then just the titles now it’s uh moving so quickly you can’t keep up it’s very hard for an individual to know what the state of the art is and it creates problems if for a couple years I’m not aware of some interesting result relevant result that uh reduces my capability of moving this whole field forward in Hugo degas’s book he talked about this concept of kind of a cyborg where you know the human would you know use kind of that AI potential agent that would work alongside of it and we’re obviously seeing developments from neuralink which is probably the closest thing we’ve seen tech-wise to that concept um we’ve seen some successes with neuralink I think one of their first patients was just talking to Joe Rogan it was really interesting uh to hear from the person who had it in their brain what he was able to do and it was quite fascinating where he was saying that he would just think about moving an appendage and then he would just actually think about something and it would be coming on the screen and even he thought that in the future he would think about a letter and it would come up and again very interesting to see that that practical evidence of that as opposed to theoretical now as I remember correctly Hugo degaris said it was a very false assumption to think you can become a hybrid because it was only a matter of time before the AI would just take over your useless body and then take Take the Wheel from there I’m curious what you think about neuralink do you see that as a potential a solution to our demise is it a distraction or does it have a possibility of us re-rooting this uh this this technology so right now it’s amazing technology I love it I support it helping disabled we can cure quics maybe blind people soon this is incredible I’m so happy he’s doing it long term if you switch from where you have this human with an assistive tool do you have this human with a super intelligent system attached to it it’s not obvious what a human is contributing so if you’re not contributing much either explicitly or implicitly you are removed from being part of that system you are not in control you not making decisions if you consuming too much uh energy you’re slowing down thinking process with your interactions you may be completely removed from it so I don’t see it as a long-term solution where we can be side by side because again what is it you adding to this equation if there is something special about humans maybe in terms of quality again only a human can experience internal States then it adds something to it a vanyan machine incapable of such uh understanding would benefit from having a humanist part of it but if it’s not the case if we can simulate fully everything brain produces in terms of a mind then it’s not obvious what uh human is doing there and uh actually the negative part is now machines have direct access to your brain in terms of guessing those internal states of fear some suffering that could be actually a very bad outcome yeah I think uh Hugo to GIS came to the same conclusion that um it was only a matter of time before they realized the human brings nothing to the party and so why are you there you’re just a drain on that on that hybrid organism um if you could get 8 billion people right now to to listen to what you had to say to change the course of this super intelligent development like what would you tell them or what actions would you ask them to do well it’s a personal self-interest issue I think all of us want to be around want to be happy want to be wealthy we can get almost everything with just narrow systems concentrate on solving real problems let’s solve aging Let’s help all the disabled so health wealth will get abundance we’ll get free labor physical cognitive let’s figure out what to do with all this free time life could be very good we don’t have to create our own Replacements so if we just concentrated our efforts we actually could do this if you got 8 billion people to come together and agree on anything yes what’s more impossible or you could at least slow it down as well which maybe is worth doing well having more time to think about it and do research uh is wonderful again I don’t know what IIA is specifically planning but I’m sure the more time he has to do it the better the outcome would be for his company does what we’ve done with with nuclear technology is that something we should be optimistic about because we took something that you know what is it 80 years ago could kill hundreds of thousands of people at a blink of an eye and today I don’t know how many nations 10 or 12 Nations only have that power and it hasn’t really yet been used on mass scale that’s that’s pretty impressive should we take you you know should should that Inspire us to know that we could all do this again with super intelligence both Inspire and scare us we know a few times we got very lucky where someone just disobeyed orders and avoided nuclear war we also know that we had nonproliferation treaty and now new countries have nuclear weapons so in some ways we failed completely in other ways we’re still here and it’s a win also you could argue that technology still complex and the resources are scarce and it’s not like you can take a laptop and get a copy of some GPT and and let it evolve it it actually has some barriers to entry or you might argue that super intelligence doesn’t would would you agree with that right now it’s still a huge project Manhattan level scale project where probably you need a nation state to do it today we’re talking about trillions of dollars but this technology is getting cheaper exponentially Maybe next year it will be 100 billion then 100 million eventually you can do it with access to some University Computer even a laptop and that’s super intelligence that you’re referring to that’s training very capable models training something AGI which is automated science and engineering which will allow you to do research to develop next generation of those systems leading to ai+ plus all the way to Super intelligence and you know I’ve got two little boys they’re six and eight and I think about my teenage years and when I was having my existential crisis last year I was literally thinking about their teenage years that were going to be you know in 8 to 10 years and thinking all the things that I used to do and you know my first car and all these things and I thought it’s very unlikely they’re going to have a reality anywhere near what I experienced um do you think about that and how do you get your head around that and do you know people that agree with you I think about it I talk about it with my kids uh I definitely know their future is going to be different from what we experienced how it’s not obvious a lot of it is up to us but it’s not going to be the same okay and so we’re mid 24 you know maybe we’re looking at two to three years before we see something that’s qualifies as AGI I think that’s probably reasonable um I know I asked you earlier but maybe I’ll ask you kind of as we conclude I mean I guess we don’t know how this plays out but you know you alluded earlier to the fact we could see these kind of 70 80 90% unemployment pieces you know it feels like there going to be these kind of major events that’ll start to really open people’s eyes probably like Hugo said they’ll be adopted by politics and they’ll be the push back um you know have you thought even to yourself what some of these events might look like you know whereas you know this happens and this happens and this happens where it’s clearly artificial intelligence almost gone too far where the public starts to wake up you must think about that sometimes and and if so you know what do you think some of those events might be so I think about it retroactively I think of 10 years ago somebody told me that we have ai as capable as the latest model whatever you want the model to be doesn’t matter pt4 CLA doesn’t matter I would tell you we definitely already have it we have a GI it can do programming it can do translation it can do all those amazing things and so the world would completely changed by that but there is a difference between having access to technology and actually deployment of that technology we saw it with video phones I think AT&T had video phones in the 70s nobody had them nobody could afford them it took until iPhone for everyone to do video calls this could be the same it could take a long time to actually deploy this in society even if we create a super intelligence for strategic reasons it can decide not to act in any levent ways for a long time accumulate more resources have more of dominance more strategic Advantage so it easily could be decades before anything really bad happens H or it could happen in a weekend maybe it already happened and we’re in a simulation I watched Terminator 2 recently through the lens of post chat GPT and uh there’s a point where they talk about where Skynet kind of like I kind of got intelligent and then it was like a certain number of days later three days later it like finally shut everything down and it was just a reminder of exponentiality and how just we really can’t conceive of how fast something could develop and grow and learn and then be over so you know yeah I I appreciate what you’re saying it could take decades but it also could take microseconds and we’re not sure what to look for if there was super intelligence in the world right now ing in this world how would we expect it to be different what changes would we expect to see immediately that’s what makes it so hard to test debug monitor those systems which is why we could be in that simulation right now we probably are but you really don’t think we are do you I wrote a paper about how to escape from a simulation so I take it seriously enough um how do you escape from simulation well I’m still here so clearly it didn’t work um doctor uh I really appreciate your thoughts on this um I really appreciate the fact that um you know you unabashedly assert that this is happening and uh I think more people need to be talking about this um and so I hope people listen and learn and take what action they can um because it really feels like that movie right now don’t look up and nobody’s looking up um every now and then I look up then sometimes I actually find myself not looking up again and so I can see my own Humanity in myself that’s not rational but this is coming as they say it’s in the post and um it’s going to change this world in a way we’ve never seen before so um I hope more people listen to this I hope we can find some solutions to this going forward um appreciate your time if people want to follow you learn more about you and uh get a copy of your book what’s the best way I’m on Twitter I’m on Facebook follow me on social media please don’t follow me home very important uh book is on Amazon there you go uh it’s called AI unexplainable unpredictable uncontrollable uh doctor thanks so much for your time really appreciate it and uh thanks for telling us what uh the future could look like thank you so much for inviting me I hope we can do it again in two years me too all right appreciate you thank you so much we’ll see you next time on London

FOR EDUCATIONAL AND KNOWLEDGE SHARING PURPOSES ONLY. NOT-FOR-PROFIT. SEE COPYRIGHT DISCLAIMER.