In this special cross-posted episode of Doom Debates, originally posted here on The Human Podcast, we cover a wide range of topics including the definition of “doom”, P(Doom), various existential risks like pandemics and nuclear threats, and the comparison of rogue AI risks versus AI misuse risks. 00:00 Introduction 01:47 Defining Doom and AI Risks 05:53 P(Doom) 10:04 Doom Debates’ Mission 16:17 Personal Reflections and Life Choices 24:57 The Importance of Debate 27:07 Personal Reflections on AI Doom 30:46 Comparing AI Doom to Other Existential Risks 33:42 Strategies to Mitigate AI Risks 39:31 The Global AI Race and Game Theory 43:06 Philosophical Reflections on a Good Life 45:21 Final Thoughts
Introduction
welcome to Doom Debates one of only two podcasts in the world that communicates that P Doom is high the second one being the for humanity podcast with John Sherman highly recommended today I’m cross-osting my recently released interview with Joe Murray on the human podcast which is a show that asks interesting people about their lives and careers so in my case Joe’s questions are all about what is doom how could AI cause doom what is pdoom what is Doom Debates’s mission what is my life like what’s the best argument against AI Doom what about other risks such as viruses pandemics and nuclear doom how do I compare rogue AI risk versus AI misuse risks what can be done to stop AI doom and don’t we need to build AGI because of bad actors if those questions sound interesting to you I think you’re going to enjoy the interview just one quick thing before we get to it can you go to dunebates.com and type your email address and submit it so that you’ll be subscribed to my email list that way you know when I’m going to publish another episode and when I see my subscriber count go up for my doomde.com substack that gives me more motivation to hurry up and make the next episode so everybody wins okay enough of this pre-ep episode boilerplate you don’t come to doom debates to have to fast forward through 5 minutes of boilerplate if you listen to Doom Debates it’s because you value your time and you value information density you expect nothing less than an aspie level of information density from this show so we’re going to go straight into Joe Murray’s interview once again his podcast is called The Human Podcast if you’re interested to check it out there’s a link in the show notes you can listen to his other interviews and here we go
Defining Doom and AI Risks
so what is doom exactly people can have different definitions i think a simple definition for my purposes is if you destroy more than 99.9% of the value that humanity was going to have on its historical trajectory you just wipe it out you know you go to such a small fraction of the potential value i would call that doom and yes it’s vague but it is kind of it’s pretty easy in practice to distinguish whether hey look there’s a bunch of galaxies with what we’d recognize as ongoing life ongoing civilization whether it’s flesh and blood or or silicon or whatever it is it looks like a civilization it looks interesting it looks productive it looks fun compared to oh like it’s just gray goo or it’s it’s just darkness right so that’s kind of the distinction I talk about when I talk about like a good future versus doom the doom that you’re you’re interested in that you speak about in your podcast is is the doom really kind of specifically associated with with AI doom so if someone’s listening and really doesn’t know you know anything about this topic really can you paint a picture at all of how doom from AI might happen what I as an AI doomer what I worry about is in 10 years or in 20 years in the near future in my own lifetime I’m worried that you’re not going to see the human civilization that you expect you’re just going to see basically a wasteland where it’s just AI doing what AIs want to do simply because they’ll be the smarter species so basically they are going to do to us what we very happily did to all the other animals right the way humans have taken over Earth you know we’re causing the sixth great extinction like a million species have already gone extinct and we’re we’re still working on that right we’re still in the process of very successfully extinct pretty much every species maybe we’ll leave beef cattle alive but we’re in the process of replacing them with lab grown meat which I support by the way but the point is we are currently manhandling the earth to do what we want even though lions and tigers they have sharper claws than us they’re bigger than us you’d think that they would have their own niche but no we are taking everything as as the human species right so that the treatment that we’re giving to the entire biological ecosystem I don’t have a problem with that as a human right i think that’s fine but I think it’s an illustrative example of what’s going to happen to us when we build AI is there any reason to suspect that this analogy won’t follow in the sense that AI will maybe because it’s more intelligent or some other reason will will treat treat us differently in the way that we treat animals is there any sensible idea that way or do you not see it normally when I propose AI doom that’s a pretty obvious question right it’s like okay yeah that’s the worst case scenario is like a species that has its way with us but can’t we just like make the species good i mean we build it after all can’t it serve us and the answer is yes in principle right like couldn’t monkeys have just built humans to serve monkeys yeah in principle right you can wire up a human brain to want nothing more than to serve monkeys right but in practice that didn’t work for the evolutionary process and so the question is how likely are we to make it work the problem is we don’t really have the tools to make it work like the way we’re building AIs we’re not actually inputting a proper wish into the AIS we’re more like we’re just batting them around and letting them evolve inside their computers by reading the internet so we’re just hoping look if it reads the internet and then it just like figures out how to be intelligent and it talks to us that’s going to be enough to to keep it uh serving us like that’s what people hope you even hear them throw in this language like look it’s a line by default if it read a bunch of nice web pages maybe it’ll be nice and so now we get into the weeds of like well wait a minute what actually is going to get outputed from this process right what what kind of mindset what kind of goals does the the actor that comes out of all this what does it actually want because I think there’s two steps here like the first step is do you accept the premise that if it’s smarter than us then it can manhandle us and it can have its way with the universe that’s the first premise and then the second premise is okay given that it can doom us will it want to doom us right there kind of two steps to the argument and this other kind of key term related to doom is is p doom what does p
P(Doom)
doom yeah so p means probability right it’s a probability of doom so a lot of Us AI doomers we believe in basian probability you know like prediction markets if you go to a prediction market you can see there’s odds for everything and you can basically uh you can express how likely you think something is in the form of a probability and then you can you can give or take odds so like if I think that something has a 70% chance of happening and I look at a market and the market’s saying it’s only 55% likely I might purchase a share in that prediction market there there’s basically ways to bet on your probabilities and basian reasoners like myself people who believe and basian philosophy we just think that in order to be a rational person and reason about things coherently you have to roughly have some idea of the probability of things you can’t just go around saying like oh there is no probability like the world ending there’s no such thing as the probability of the world ending like no you you have to put some probability of it I’m not saying you have to know the exact decimal point right I’m not saying you have to have a super precise probability but if you can’t tell me the difference between like oh yeah there’s a 99% chance that humanity will survive for the next million years compared to like oh no there’s like roughly a 50-50 chance or like oh no there’s like a less than 1% chance like there’s something called you know the ballpark probabilities so when we talk about pdoom we’re just saying like very roughly are we talking like no problem we’re totally going to survive or are we talking like we’re almost certainly not going to survive and as for me I’m roughly in the middle so I I’m sometimes I say I’m 50% sometimes I quote a leader of one of the AI labs who said that he’s 10 to 90% sometimes I quote Dario Amade from Anthropic who says he’s 10 to 25% % pdoom there’s a lot of smart people who are in this vast middle range that’s definitely not like oh we’re 99% doomed but it’s also like oh don’t worry about it we’re not doomed there’s a vast middle range 10 to 90% where it’s like uh we’re kind of doomed like it seems like we very well might be doomed in our own lifetimes so that that’s my PDM is like this middleing range of like hey we really might be doomed can you explain a little bit more about how you go about determining what your PDOM is so what is what is your PDM roughly and and how do you come up with that yeah it’s roughly 50% but like I said it’s more like 10 to 90% um I think the easiest way to explain it is to be like look imagine I told you that my PDUM was only 1% that seems like I’m being stupid because like how is it only 1% even if you look at nuclear conflicts right i mean we humans we we come close to dooming ourselves all the time like the Cold War there were so many close calls it’s very fascinating to to read like Wikipedia’s list of nuclear accidents you know we literally dropped a bomb where three out of four of its safeguards we accidentally dropped it uh on on in the US in the Midwest and it almost went off but luckily the last safeguard worked because there was like a 20 volt switch that luckily stayed stayed in the correct position even though all the other fail safe like it would have been like a 10 megat ton blast like insane stuff like we came very close to nuclear doom and many many different times in the last 50 years um so this idea that humanity can doom ourselves like it’s clearly on the table it’s clearly on the table that AI might go crazy on us and it and it might u you use its intelligence to to manhandle us in a way that we don’t like so if I were to tell you a number that’s less than 1% i feel like I’d be failing very bad and similarly if I were to tell you a number that’s greater than 99% it’s like really there there are some very smart people who think that we’re not there are some smart people like Robin Hansen who thinks well AI is probably going to take over eventually but it’ll be okay because it’s our descended so just the fact that there’s so many smart people disagreeing and having different opinions makes me crazy to say greater than 99% so I’ve now given you the argument why it’s clearly 1 to 99% and then it’s just a matter of now we’re just talking numbers right like should I push it past 5 or 10% i would say definitely yes because of all these different arguments I could tell you so basically when you hear a bunch of arguments and all of the arguments sound strong and they all sound mutually enforcing and they all sound uh convergent like there’s a lot of deep principles why there’s like many different ways why they’re doomed that tends to push somebody above 5 or 10% and then on the other side well why can’t I push past 90% well the obvious argument is like we don’t want to die right so there’s a lot of humans that are highly motivated not to die and so far we’ve muddled through with nukes so I have trouble giving a more than 90% chance of doom just because we’re all going to try really hard when we get sufficiently scared so that’s roughly why I end up at 10 to 90% so you host this this great podcast called Doom
Doom Debates’ Mission
Debates what exactly you trying to achieve with this podcast like how do you think this can maybe help AI safety and riskrevention efforts so for me it’s very clear that we’re doomed right so I have a strong position that AI is totally reckless right now to keep developing further it’s like insane the grown-ups in the room really ought to shut it down problem is a lot of smart people aren’t on the same page as me so my podcast Doom debates is just systematically saying “Okay let’s bring all the smartest people in the room with all the different perspectives of why we’re not doomed or why we shouldn’t pause AI development and let’s just debate them all right like we’ll just it’s like a video game right just everybody come at me tell me why you think we’re not doomed i’ll tell you why we are doomed.” And my goal with the podcast is that viewers can just watch episode after episode okay this person has a different reason why we’re not doomed but their arguments aren’t holding up this person has a different reason their arguments also aren’t holding up why does everybody have different reasons why we’re not doomed that all seem weak so I’m hoping that somebody It’s hard for me to lay out the entire argument why we are doomed but when you see all the different people trying to argue that we’re not and they’re all kind of not seeing big holes in their argument I hope that that will together create a case of like okay um maybe by the process of elimination of all these bad arguments maybe we actually are doomed and it’s time to act so that that’s kind of the the approach that I’m taking with doom debates then higher level the the general strategy you could call it fear-mongering what I’m doing I actually think fearmongering is good and the reason is because we’re building mutual knowledge that we’re doomed mutual knowledge meaning we need to know not just that we know we’re doomed but that I need to know that you know that I know that he knows that everybody knows that we’re doomed or that the majority of people know that we’re doomed the reason it’s mutual knowledge is is because individuals already suspect that we’re doomed like if you go to a random person in the world and explain the very basic situation with AI like it’s getting smarter than us we don’t really know how to control it how does that make you feel and they’ll be like bad i don’t like it i would rather stop it it’s like great you think that most people think that most people don’t realize that most people think it and think that it’s urgent so I’m just building mutual knowledge interesting so you think a lot of people if they were to just take a few seconds to hear some arguments and think it through would actually be already you know you know possessing a similar belief to your to yourself exactly most and you know surveys show there haven’t been a ton of surveys done but there’s been like four surveys that I’ve seen done on on the American population there’s been some outside the US um you know I don’t have the latest data but I’ve never seen anything other than a result saying yeah the majority of people don’t feel that we need to race to AI right now they’re scared of the consequences they’re scared of human extinction and they they want to see regulation like this is a popular opinion so the funny thing is I’m normally a contrarian i’m normally in a position of like look people need to learn they don’t understand what’s good for them let me teach you what’s good for you but here I’m more of I’m a contrarian against the tech industry where it’s a bunch of engineers who are going to their job making these high salaries building technology that they think is cool and I have to explain to the engineers of like hey what humanity wants what most of your peers want who aren’t engineers like you is they don’t like what you’re doing because as your own CEOs admit it’s reckless it’s not something you actually know how to control you just like it and you’re making money but they want you to stop so that’s that’s what I mean is like the majority of people are on the same page but they don’t really have they’re not unified to take political action because they don’t realize how urgent it is so it was basically my job is to be like “Hey guys wake up.” Like you just need to wake up once you wake up you already know what to do you already know this is bad and so when I’m fear-mongering I’m just bringing attention i’m like “Hey look this is your leaders right now they’re dooming you like do you guys want to do something politically?” That’s basically my position so what was your what was your journey into hosting this podcast yeah so my my journey began when I was in college so this is around 2007 um I I’d always grown up as like a software engineer love programming and I considered myself rational but then I read Alazarowski i read Less Wrong and I’m like “Wow this guy has a lot of really powerful philosophical arguments arguments about how to think better and also arguments why we don’t know how to control AI.” And that’s going to be really scary when when we build AI and we’re probably going to build it in an unaligned uncontrollable way were probably going to mess up the security so I kind of went down the entire rabbit hole all the way back in 2007 and then I I I was so moved by it that I went and met Aliarowski in person i donated a lot of money to the the machine intelligence research institute that he was running and the center for applied rationality so I’ve I’ve been steeped in this stuff for 17 years um only thing is I used to think that it would take a long time right so I used to be like “Wow these ideas are just amazing they’re way ahead of their time so like in 2100 right these are really going to come into play.” And then the problem is as we all know a few years ago you know with GP3 GP4 we were like okay so the time is actually now right like life comes at you fast and to uh to Kurtzw’s credit right he’s always been saying 2029 to super intelligence so you got got to hand it to Kurtzswell like he he wasn’t surprised but I was surprised that you know that it only took until the 2020s to get to the touring test to have a bunch of people now predict that we’re going to have AGI in 2027 like this is like the new ballpark like 2100 now seems very late and so now I’d spend my whole career just being like whatever I’m just going to be a software engineer i’m just going to do startups but now I’m like well I I can’t look away from this right we’re all we’re on a conveyor belt that’s taking us right to the whirling razor blades right starting another company right now doing anything where you’re just focused on like long-term financial success it’s just crazy it’s like you know it’s it’s it’s a it’s a train wreck happening right now um so in terms of my podcast you know I’d built up a following on Twitter um I enjoy tweeting a bunch of stuff like clips of people saying dumb stuff on podcasts about AI or people saying smart stuff on podcast about AI so I noticed myself spending a lot of time tweeting and and building up Twitter followers um and then finally I was like look if if I’m spending so much time producing this content why don’t I just you know maybe I have the stamina to produce episodes of a podcast where it’s like me talking and I interview a guest like let let’s try it that was basically the threshold I needed to reach because I’m like look podcasts are a lot of work right why would I have the stamina it’s just me noticing how much I was tweeting i was like “Okay tweeting is a waste of time let’s convert the same energy to a podcast.” And sure enough I’ve been doing it for 6 months and I still have the stamina to do it and I’m having a lot of fun and it’s growing so that’s basically been my journey to Doom Debates given your concerns and your worries are on AI like how are you how are you doing personally how do you feel
Personal Reflections and Life Choices
about all this i feel fine it’s just you know people’s feelings are are mostly just genetic I think right so if if somebody has like a predisposition to be depressed then they’ll notice AI and they’ll be like “Well I’m depressed because of AI and they’ll be logically correct but it doesn’t really help to be depressed.” Whereas um you know I’ve been fortunate i don’t suffer from depression so I have like a decent happiness set point so you know every day I wake up and I’m like “Yeah humanity is doomed um that doesn’t make me feel too depressed.” So and it just makes me like “Okay well I guess I’ll just spend my time working on Doom stuff.” That’s kind of like you know I just feel like that’s like a good niche for me of stuff i could work I could produce good content on it that’s fun for me too um you know I’m getting some daily enjoyment for that so I feel fine um I think psychologically analyzing myself um one thing that probably makes me feel better is that which isn’t even rational but it’s like when I think about doom it I don’t think of like myself getting screwed the fact that everybody else gets screwed even though it’s my own family i’m just saying psychologically I think if I do some internal reflection I think the reason it stings less for me is I guess there’s no like social status component if anything my status goes up because like look I called it i told you guys right i guess so I’m I’m basically one of the coolest people in like the last hours of the world ending because I I was one of the people who who told you guys that the world was going to end but you know me and my family living like a caveman that’s that’s not going to be fun right so I’m not saying I want that to happen i’m just saying intuitively why am I not literally depressed right now it’s probably like those those status dynamics being uh computed by by my subconscious yeah you talked about how your your kind of career choice now has been you know influenced by your beliefs in AI like you’ve got to do this podcast you’ve got to think about this issue is there any other aspects of your life that you have changed given your beliefs about the future perhaps regarding money or any other thing so the aspects of my life so number one I’m not interested in doing another startup or angel investing for the long term those are activities that I used to do right that used to be fun for me and now it’s just like what’s the point of a long-term payoff like you know it’s it’s certainly changed like how I invest because it’s more like I don’t need to optimize my life for trying to get richer and richer until I retire if I retire on welfare like that’s still a win for me you know like or if I have like you know if I can just keep my current house that’s already a win like the fact that I even get to retire at a normal retirement age right which for me it would be in like I don’t you know 25 years um that’s uh that’s a huge win right so it’s it’s totally changing the goalpost for me um the other way to change it is I I’m living my life on a 50/50 like my life is going to branch out so yes I’m still saving some for retirement right i’m not totally screwing my life i’m not taking up a smoking habit right i’m not doing things that are like terrible for the long term i’m still exercising um but at the same time if there’s things that I can frontload to get more value out of today then I totally do so I tend to I’m spending more today than I than I otherwise would for example um I’m saving less i’m not not saving but I’m saving less and you know I’m having kids i have three young kids right now um and the question of whether to have kids it’s definitely weighing on me like what’s the point of having kids when a lot of the value of kids is when they grow up and as so that you don’t have to like change their diapers all the time and yet they’re not going to have much time to grow up um but I’m just I’m really hoping that the 50% good scenario will happen right so I’m trying to make a lot of the same preparations as if I was more certain about the good scenario because I think the good scenario might happen i just think the only way it will happen is if we pay attention to it and actually do something to stop it which we’re not right now right so I’m living two simultaneous futures right now it’s interesting to hear you then say about 50/50 because I was going to ask again like are you 50 are you 50/50 in the middle because you’ve said a lot on this episode that we’re doomed and not like we might be doomed and I was thinking are you a little bit more on the side that things will go wrong but you think you’re kind of more in the middle is that right i’m in the middle because look everything we’re doing and almost everything we’re doing as a society is we’re acting like a society that gets doomed and and it’s not it’s not because we have like a death wish it’s because a lot is required of us to not be doomed and we’re not stepping up like not at all like we’re not even there’s a big checklist of things we would need to do if we wanted to not get doomed unfortunately you know reality has given us a brutal checklist of things we have to do if we want to not get doomed and we are barely scratching the surface of the checklist and time is running out so that’s the reason I say we’re doomed now I still hold out hope that at some point we’ll get sufficiently at some point in the next few days or whatever however much time we have we’ll suddenly realize like okay we got to break out the checklist we got to do all of these things on the Jack list like stop AI uh don’t you know augment human intelligence so that at some point we can align AI you you know we have enough intelligence to solve the problem because even the smartest humans today are struggling with even knowing what alignment success looks like so like all these different things we would need to do and like you know enforce it properly um don’t have uh leaders of companies being able to influence the government so much about why they need to build AI so like all of these different things we would need to do if we wanted to have any chance of succeeding maybe we’ll start doing them in the future but like time keeps passing and and we’re not doing them so but when I say there’s like a 50% chance that somehow it’ll work it’s basically a combination of like number one I’m insane and I have no idea what I’m talking about and like look at all those guys saying AI is perfectly safe right like look at Yan Lun saying there’s nothing to worry about look at Robin Hansen look at Mark Andre all of those guys are smart and I’m stupid right so that’s part of my 50% probability of how we’re going to survive uh and another part of the 50% is like “No I am smart but society is going to get smart fast enough so that we don’t kill ourselves.” Like that that’s that’s basically those are those are basically like the two and then of course there’s like there’s a little bit of uncertainty of like okay I’m smart i know what I’m talking about but like for some reason there’s just going to be some big AI winter where like the last part of intelligence for some reason is going to be hard so there’s like a little little bit of probability for that but yeah I mean that’s it it’s it’s not really a good story like it’s if if I just think about how things are going to go they’re probably going to go very badly yeah you mentioned three people there like Mark Andre Yan Lun and um Robin Hansen robin Hansen if if if either of these were did if all of these people did turn out to be correct that things go fine things don’t go wrong what would you guess might be the reason that that happens is there any argument that ever slightly tempts you to think you know what that’s I really think things are looking pretty bad here but I’m slightly open to this one this could be that this is the kind of strongest counter argument I hear well as I mentioned earlier they all say different reasons why we’re not doomed right so they’re not even getting their story straight among themselves um the Mark Andre argument is the easiest to dismiss because he rejects like very very basic stuff like he rejects the orthogonality thesis the idea that arbitrarily smart AIs don’t necessarily have to be moral his perspective on that is like no no they’re they’re totally look you can talk to them today they’re moral they respect human values so I guess he could in principle be right and arbitrarily smart AIs could totally deeply internalize human values and and things go right for that reason from my perspective we’re fooling ourselves to think that they really get human values just because they’re chatting with us about it um but yeah if Mark Andre is right about that which he certainly doesn’t talk like somebody who’s studied the issue enough to be right but if he happens to be right anyway then great then we’ll have like this great moral genius AI who is the caretaker of humanity you know basically like a creating a good god great that is logically possible like I said it doesn’t look like that’s what we’re actually creating but it maybe in Mark Andre’s mind it is and if he’s right then great i hope he is um if Robin Hansen is right his his perspective is a couple things um he’s basically saying look there’s going to be time we’re going to have plenty of time to see if there’s any issues and react to them and maybe we can even stop building it if we see an issue whereas my perspective is like time like what are you talking about like people are predicting there’s like two years like where are you seeing all this time and he’s like no I’m skeptical about the 2-year prediction i think it could take a 100 years so like I said so yeah so taking a long time is one way we could potentially not die um and then um young lun his argument is like look people aren’t going to allow a dangerous AI to get released like sure the current architecture would be dangerous but we’re going to have like a different architecture that people will figure out a way to make it controllable before they release it so he’s kind of optimistic about like the process like the AI building and release process whereas from my perspective it’s like really the process like we’re just recklessly building it and releasing it and just and hoping it works i don’t really see optimism on the process side so those those are different arguments you can make where you could be more optimistic than me but like I don’t see it i’m interested like who you’d really love to see have a debate and discuss this more perhaps with yourself or perhap between themselves and maybe these are some of the kind of techy people we’ve just mentioned or maybe it’s government leaders the president and so on like who do you hope on the world stage now has more of a discussion and an open debate about these about these issues like who’s missing from it
The Importance of Debate
so first of all I support debate in general so when I talk about doom debates doom doom debates has a twofold mission you know my show doom debates uh number one of the mission is raising awareness of AI doom because like I said before we just need to fearmonger we just need to build mutual knowledge we don’t even have to convince people we just need to build knowledge so so that’s that’s uh mission number one of Doom debates is the awareness raising but to your question the second part of Doom Debates’s mission is to build the social infrastructure for high quality debate this is surprisingly missing right now like you think of modern civilization as like ah yes debate it’s a practice that goes back to the ancient Greeks right the Greeks and Romans like debate is a respectable enterprise but if you actually think about it besides like a presidential debate or an occasional political debate like when do we actually use debate as a tool in our society well there was the monk debate uh in 2023 that brought together Yan Lun Max Taggedmar Melanie Mitchell I think Yosu Benjio was the fourth one so once in a while we have a debate whereas from my perspective if you look at somebody like a Mark Andre or a Sam Alman a Dario Ammedday a Demis Hassabis all of these AI leaders right sati Nadella from Microsoft all of these AI leaders they’re always just going on their own friendly podcast making their own pitch why don’t they come and debate why don’t they debate one another why don’t they debate alias for Ukowski because right now you have a phenomenon of dueling podcasts right everybody just gets to broadcast their own message where is the ring where do we hash this out where do we disagree in a productive way so Doom Debates you know the goal of Doom Debates is that a year from now or whenever we get to an audience of a million listeners right because that’s the goal is grow the audience then people will expect these debates to happen so I could be the moderator I could have different guests on whatever it is these debates need to happen and to your question of like okay who should debate well right now I think that the leaders of the AI companies should be made to debate but I also think uh people governing policy should be made we should have policy debates even without involving the AI leaders because at the end of the day the AI leaders are like tobacco company leaders right like it’s it doesn’t it doesn’t matter what they think because they’re being bad so if they will never agree that they’re being bad we still just need to regulate anyway because they are being bad right like they need to stop regardless of whether they think they need to stop so just go back to a couple more questions about yourself what is it about the way that
Personal Reflections on AI Doom
you think maybe your personality and so on that is has led you to be the guy doing this like why are you see why do you believe you’re seeing something that basically no one else in public is really paying attention to or thinking about so in 2025 it’s it’s a little late to say that nobody else in public is is thinking about it or paying attention to i mean there definitely is a ground swell right i mean 2023 was really the end of the era where you could be like “What a weirdo you are as as an AI doomer nobody’s saying this.” And believe me people tried right so in early 2023 when uh when GPT4 came out and when AI started really making a splash in the news like “Oh wow this is going to transform the economy.” And Sam Alman was going around saying um yeah the good scenario for this is heaven but the bad scenario is basically lights out for all of us sam Alman back when these AI lab leaders when they weren’t getting so much attention and so they felt like they could speak more freely back in 2023 a lot of people tried to character assassinate doomers and portray doomers as being weird and out of the mainstream and it used to work because it used to be the case that like yeah you don’t really have surveys right so and and nobody nobody ever talks about this but since then we famously had the Center for AI Safety they did this pause letter which was a one-s sentence statement and it basically said “We should take AI risks as seriously as other existential risks such as nuclear and biological threats.” So very simple statement it was literally one sentence and pretty much everybody signed it not literally everybody like Yon was a notable exception but most of the CEOs of AI lab signed it sam Alman signed it dario signed it dennis Zava signed it and then if you look at Turing award winners right so Yashu Benjio Jeffrey Hinton signed it so like the the biggest experts in machine learning signed it um Elon Musk I think signed it or he signed something very similar if it wasn’t that um Daniel Dennit the philosopher signed it i mean the list goes to hundreds of names it’s hard to think of somebody prominent in AI who didn’t sign it so this built what I’m saying mutual knowledge this was major major mutual knowledge so credit to Dan Hendris for putting this together because this blew the Overton window doors open and so now when we talk about AI risk it’s not like hey trust me I’m a doomer it’s like um do you see this statement like this is what smart people think this is actually a consensus it’s just we need to drag along like it’s just a matter of time before this diffuses to everybody else’s opinion because it jibes with what people think for themselves like hey when I think about AI am I scared yes are these luminaries scared also yes right so it’s just a matter of time so when you ask your question of like what you know what do I think makes me qualified well first of all there’s a lot of people like me okay but maybe I can interpret your question as like well why am I making a podcast like why you know why why am I doing this uh why am I dedicating my life to this instead of like having a job and just like reading the news and I guess the answer is like I can’t look away right so I just I just feel like this is the only thing that I would be able to focus on with a lot of my time and then the other reason is like I just you know I feel like this is um a good fit right like this is where I can combine some of my talents and some of my background right it’s like I know about AI Doom i uh am like I’m like an entertaining speaker relatively speaking right so a lot of people who are like uh nerdy like myself and really into technology they tend to just not be very entertaining speakers I think I’m on the intersection of being like a pretty entertaining speaker and so the idea of like doing a show and building attention I think is a good fit for me have you ever worried about something else as intensely as this before like risks before you kind of nuclear risk or anything else before you kind of started reading up so much on AI AI doom is the first time that I’ve really been like hey something’s going to doom us so it’s
Comparing AI Doom to Other Existential Risks
not like I’m a lifelong doomer I’m certainly not depressed and that gets to the point about like people trying to character assassinate doomers and actually Sam Alman this was very frustrating but Sam Alman tried to character assassinate Aliowski one of the most prominent doomers Sam Alman went on Barry Weiss’s podcast I think and and he was like yeah Aliowski I like him but he’s kind of a prophet of doom he’s always saying that things are doomed which I thought was such a low blow so Sam Alman is uh full of lowb blows from my perspective um so neither I nor Alazarowski are profits of doom we don’t go around saying things are doomed if you look at climate change I think it’s important to try to slow global warming but I don’t think it’s something that’s going to be an existential risk i think it’s manageable we can pump sulfur into the atmosphere it’s even the worst case scenario of climate change is probably like okay we took 10% out of the value of the economy like 100 years from now which and 10% is a lot right it’s it’s it’s like a lot of human lives but it’s just 10% um nuclear would be I think by far the biggest doom risk u right below AI risk so nuclear risk when I think about the chance of uh nuclear war setting back civilization like a thousand years that chance for me is something like 1% per year so if you add that up over a century it’s like more than 50% so I do think nuclear doom is very serious and I am a nuclear doomer but when you look at AI doom I’m talking about 50% in like the next 15 years so that’s like 10 times more doomy than nuclear doom so if you’re a question about like where the doom is it’s like really mostly AI doom but also some significant nuclear doom could you put a P doom on nuclear doom yeah nuclear doom is something like 1% per year at this rate in the next century it might be something like 50% for completeness by the way I should add you know uh pathogens like biological virus doom I think is very high especially with AI i think AI makes it very easy to design very deadly viruses so I do want to throw in uh uh virus doom um a little I would slot it in below nuclear doom so maybe like a third of the risk of nuclear doom you also want to attribute to viral pathogen doom but it’s just AI doom is is just is the gorilla in the room here like it’s nothing else comes close are you more concerned about AI misuse risks or kind of rogue AI ai going off on its own or equally worried or how do you see the distinction here and stuff my biggest concern is rogue AI um when I think about somebody misusing AI the only difference between that and a rogue AI is like the the first command so it’s like okay do a bad command boom compared to try to be good boom like the boom is the dangerous part the fact that we’re creating this this this thing that can do a boom the fact that we’re creating this thing that can go off and extinct us if it has the right trigger i’m not so concerned about like oh my god is there going to be a trigger the fact that there can be a simple trigger basically means yes like I don’t think we’re going to avoid setting off a very simple trigger if we build a trigger if we build a triggerable system you touched on this a little bit earlier but if we could to hear a bit more about
Strategies to Mitigate AI Risks
what you think could be done exactly to you know significantly reduce the chance of doom like what what do we need to see worldwide from governments from companies from individuals maybe from podcasts from all these kind of different groups to produce a successful outcome so I think we’re very far from a success welcome we don’t look like the society that’s going to have a success welcome um we would have to make a desperate effort to have first of all have mutual knowledge of like yes this is very scary yes you and your loved ones could actually die very soon so so first of all mutual knowledge i don’t think this can be done without that mutual knowledge of like hey we’re on the brink here like we’re about to lose control once we have that mutual knowledge then I think that reframes everything because when you have somebody like a Sam Alman or Dario Amade saying like trust us we’re just going to go build the next model and then we’re going to make it safe everybody would then viscerally scream like stop step away right like it’s almost a little bit like the disgust reaction you might have if somebody’s like “Listen I’m just going to make a human clone i just want to see what this baby does.” Which is much safer than than uh than AI that people might have like a visceral reaction of like stop messing with God don’t touch that or like whatever it is or somebody’s like hey I’m just going to play with this plutonium right in my lab it’s like what are you doing uh so whatever analogy you want to make like we need the shared mutual knowledge that would reframe what everybody’s doing like these software engineers go going to their job getting their paycheck it’s like what are you doing what are you doing this is completely insane once we have that then it suggests policies very similar to what we’re doing with nukes like okay this all needs to be regulated maybe all the different governments can all collaborate on their own Manhattan projects that are all controlled in a secure data center with like the best practices of of how to try to build this with like the best safety engineers maybe we can do that we can compromise and we can do that as opposed to not developing super intelligent AI at all but the idea that labs can just develop unilaterally on their own hardware without central oversight it’s just reckless and that’s coming from me i mean I lean libertarian it’s not like I want to centralize things i just think we’re building something ridiculously reckless so total you know doing something as intense as the way we lock down nukes and really more intense that’s roughly what I think we should be doing and it gets to the limit of what I can personally recommend like I’m not an expert at this stuff i’m not an expert at how to centralize AI development i just know whether a civilization looks like it’s trying to address the problem or not so I can vaguely point you in the direction of what it looks like to try to address the problem you said there maybe something similar to nukes being locked down can you say any more about that at all about what you’re envisioning should happen to AI is this like a banning it is this pausing it like what does this involve i basically mean a pausing frontier AI development right now i think it’s insane for any AI company or individual or any entity really to to be creating a smarter AI than what we already have i think it’s dangerous because like do I think in the next year we’re going to be doomed no but I think the threshold is unknown it’s like we’re so you know we’re skiitting right there’s a slip slippery slope i think we’re skiitting right now and if I wish I knew exactly how far we could skid before we need it to stop right i wish I knew but nobody knows right so it’s very possible that the next time that one of these AI companies trains their next model with whatever secret sauce they want to give it well the next time they do that it’s possible that you get recursive self-improvement you you get this breakout it turns into like a virus where the AI is like “Oh you ask me a question you ask me how to make money for my business here run this script.” And then the script is like “Yeah I’m going to copy myself i’m going to be a virus the virus is going to be super intelligent.” And suddenly you already have no off button like it’s it’s very easy to imagine we’re just a year or two away potentially from a script being generated by these intelligences that starts bootstrapping it just becomes unstoppable so like the unstoppable condition is not very far away like a virus that you can’t clean out is not hard at all to imagine like there’s been many historical incidents of dumb viruses being ridiculously hard to clean out so having an AI that just permanently runs away from us the way CO ran away from us right like there’s no hope of eliminating CO right now right so so co is now just part of our uh of our whatever you call like the viral ecosystem like that’s that’s kind of here to stay similarly we’re I think we’re getting close to an age where we’re going to have super intelligent AI viruses just living on the internet you know slowing down data centers left and right helping themselves do a lot of resources i I think we’re close to that threshold already and that’s just the beginning like that’s just to give you an intuitive sense of like how things are going to start jumping um so yeah and and from there you don’t know how quickly they could be like “Oh I have all these resources let me start self-improving let me start making successful versions of myself and suddenly you get the singularity right you get the what we call the foo like things just go so fast like a nuclear chain reaction because software moves very very fast like iterations of software can move very fast so anyway so getting back to my point of what we need to do right now we’re skitting we just need to stop skitting and if we want to keep skdding if we want to keep trying to uh push our luck let’s at least do it in a centralized way with practices that we’re actually doing best practices in terms of security like it’s not connected to the internet for god’s sakes right today all the different AI training that’s happening probably has a short path of the internet if it’s not literally connected to the internet there’s it’s probably realistically has some signals that are connected to the internet or like some USB drives get plugged in regularly to something that’s connected to it like even just that like the most basic kind of stuff we’re not even doing consistently so basically a centralized project with centralized controls with a world government you know world treaty ability to be like um you’re violating the treaty you can’t do this right to arrest people to asari once pointed out if there’s like a rogue data center that’s in violation of the treaty you need to be able to bomb the data center or you need to be able to somehow you know bring the data center to justice that’s the kind of thing I have in mind if we are to have a hope of surviving okay on to a couple final questions well I’m about to ask one now that I think is you know becoming like a bit of a cliched question but I do think it’s important i’m intrigued to hear your view on it if someone says well we need to you know perhaps in particular countries perhaps at a particular company we need to develop AI before bad actors do this is the reason we need to do it um what do you think about that so
The Global AI Race and Game Theory
let’s think about the game theory payoff matrix right so normally you’re like it’s kind of like the prisoners dilemma people assert that it’s like well I’m in the prisoners dilemma and like yes it’s nice if we both cooperate but I think the other party let’s say China right that’s the most frequent example the other part is going to defect Therefore I need defect you know the defect defect equilibrium is going to be better than the than cooperate defect and that’s why I have to defect um here’s the thing it’s a different payoff matrix so let’s say it’s the US versus China if the US and China both don’t build AI then we hopefully don’t get AI doom if one of us builds AI we get AI doom if both of us build AI that doesn’t save us from the AI deal that just gives us double AI doom so when people talk about beating China they’re already living under an assumption that there’s some significant chance that we’re not going to get doomed and instead things are going to be good so they’re kind of saying like look as long as there’s a significant chance things are going to be good let’s think about the payoff matrix when things are good if things are good then whoever builds good AI is going to be better off we can’t let only China build the good AI we have to build the good AI so the whole discussion in my mind is irrelevant if there’s a very small chance of good AI and a very high chance of building the AI that dooms us so when people want to have that discussion they to me it’s it just seems like they’re interested right that discussion humans are always gravitating toward discussions where it’s like me versus them right us versus China let’s let’s have the slap fight this is very tempting right any any discussion you ever try to have with people always gets derailed toward like how can I talk about like social fights right and by the way this is there’s a connection between this and character assassinating so you try to have a discussion about AI doom very quickly it devolves into discussion of like why are doomers like this why would they be so pessimistic why would they be doomsayers right because it’s just always tempting to derail every discussion and go into like humans fighting other humans in the case of um of building AI it’s very tempting to be like oh is it going be a good AI or bad AI doesn’t matter because China’s going to build it and the US is going to build it and who’s going to build it first but it’s like wait wait wait wait wait the AI is going to do us like keep that in mind which completely makes it an irrelevant payoff matrix so you think it’d be it’s a safer approach to try and convince all parties not to build it than kind of oh they’re going to build it let’s build ourselves safer this would be a better thing if we can spread the message and get no one to build it before it’s it’s safe to do so yes because I I just want to point out when any whenever anybody’s talking about a race they’re preassuming that there’s a prize at the end of the race instead of doom at the end of the race now I’m not saying that that’s impossible there might be a prize but the whole argument rests on the probability of it being good versus bad i claim the probability of bad is very very high right now and we really need to just focus on the probability of bad rather than just praying that it’s good and then having the best strategy if it’s good i just don’t I’m not convinced by the prayer that it’s good i think it’s very clearly bad right now in the background I’ve just been interested the whole conversation is that a photo of Richard Fineman yeah exactly it’s a photo of Richard Fman my wife got it for me for my birthday cuz she knows that I like Fineman um and he was one of the rationalists um before the modern rationalist era so he was a couple generations before the Aliar Uttowi era basically and he has a lot of rational insights that he discovered like intuitively and by reading other scientists and he’s really great i mean I like to think of modern rationalists as as following in the tradition of people like him and ET James and Carl Sean so I I feel like they were like the ancestors of modern rationalists so I ask everyone the same question to finish finish my episodes later on so last
Philosophical Reflections on a Good Life
question is what do you think it means to live a good life so that is definitely a question uh you know asking that question that’s like the outer alignment problem right right so if I actually had like oh here’s a specification for you of a good life that would solve half of the AI alignment problem because half of the problem is specifying what’s a good life and then the other half of the problem is knowing how to make an AI take the spec and run with it that’s called the inner alignment problem and actually both of those problems are very unsolved right now so you’re basically saying hey how do you solve the outer alignment problem and specify a good life um it’s you know we have we have some threads to go on so for example like I think that a good life should have some amount of problem solving to it some amount where like you have to like you know work work on something and uh solve like a puzzle or a challenge right and and you have like the possibility of failing and then it’ll feel good when when you succeed i think that um as as an ancestral human right as as as somebody who has human values I think I appreciate that component of things and then of course there’s like the sensation of happiness or just like positive sensations right feeling energized feeling good right so whatever qualia is I think you want to mix that into a good life do you want to go all the way to wireheading 24/7 probably not so it’s pretty funny i mean I just listened to a couple podcasts where people ask a similar question of like what’s a good life and people try to sound profound of like oh I don’t know for me it’s like making a contribution and I’m like “No man you just asked for a spec for heaven okay let’s not try to oversimplify the question because it’s precisely oversimplifying that question.” Which is why people underappreciate how screwed we are when the AI becomes super intelligent and then doesn’t actually give a crap about what we think is a good life and just like gives us something much worse like oh you guys said that you liked happiness okay I’ve I’ve now uh edited your your gene so that you just always walk around with a smile all day done it’s all done and then really we’re like in cages or whatever so yeah i mean I I don’t have much to say besides like here’s a bunch of good components you want some mix of them and then that’ll make your life better rather than worse okay I’m getting from you this is a this is a tricky question then yeah exact it’s definitely a very tricky question and unfortunately we’re about to run out of time to solve it before you know the AI is just going to then decide what it thinks is a good life or what it thinks is a good universe and that will be that appreciate the honesty on that question yeah Lyon thanks so much for taking the time to speak and and uh been enjoying your podcast so good luck with it and Final Thoughts good luck with your efforts to uh bringing about a safer future appreciate your time today yeah Joe appreciate you as well thanks for doing this i hope you enjoyed today’s interview if you did please support the channel by hitting the subscribe button below see you next time