ROMAN. I’m looking at problems which are likely to happen and it’s not just me saying it. We have Nobel Prize winners, touring award winners all saying this is very dangerous. 20 to 30% P(doom) this is standard in industry. 30% is what surveys of machine learning experts are giving us right now.
JOE. So what is worst case scenario? Like how could AI eventually lead to the destruction of the human race?
ROMAN. So you can asking me how I would kill everyone. Sure. And it’s a great question. I can give you standard answers. I would talk about computer viruses breaking into maybe nuclear facilities, nuclear war. I can talk about synthetic biology, nanotech, but all of it is not interesting. Then you realize we’re talking about super intelligence, a system which is thousands of times smarter than me. It would come up with something completely novel, more optimal, better way, more efficient way of doing it. And I cannot predict it because I’m not that smart.
the Joe Rogan Experience. So, this is logical. And the problem is, like I said, when I’ve talked to Mark Andre and many other people, they think this is just fear-mongering. We’ll we’ll be fine. This is worst case scenario. We’ll be fine. It is worst case scenario, but that’s standard in computer science, in cryptography, in complexity, and computability. You’re not looking at best case. I’m ready for the best case. Give me utopia. I’m looking at problems which are likely to happen and it’s not just me saying it. We have Nobel Prize winners, touring award winners all saying this is very dangerous. 20 30% pdoom this is standard in industry. 30% is what surveys of machine learning experts are giving us right now. So what is worst case scenario? Like how could AI eventually lead to the destruction of the human race? So you can asking me how I would kill everyone. Sure. And it’s a great question. I can give you standard answers. I would talk about computer viruses breaking into maybe nuclear facilities, nuclear war. I can talk about synthetic biology, nanotech, but all of it is not interesting. Then you realize we’re talking about super intelligence, a system which is thousands of times smarter than me. It would come up with something completely novel, more optimal, better way, more efficient way of doing it. And I cannot predict it because I’m not that smart. Jesus. That’s exactly what it is. We We’re basically setting up a adversarial situation with agents which are like squirrels versus humans. No group of squirrels can figure out how to control us, right? Even if you give them more resources, more acorns, whatever, they’re not going to solve that problem. And it’s the same for us. And most people think one or two steps ahead. And it’s not enough. It’s not enough in chess. It’s not enough here. If you think about AGI and then maybe super intelligence, that’s not the end of that game. The process continues. You’ll get super intelligence creating next level AI. So super intelligence plus+ 2.0 3.0. It goes on indefinitely. You have to create a safety mechanism which scales forever, never makes mistakes and keeps us in decision-m position. So we can undo something if we don’t like it. Then it would take super intelligence to create a safety mechanism to control super intelligence at that level. And it’s a catch 22. If we had friendly AI, we can make another friendly AI. So if like aliens send us one and we trust it, then we can use it to build local version which is somewhat safe. Have you thought about the possibility that this is the role of the human race and that this happens all throughout the cosmos? is that curious humans who thrive on innovation will ultimately create a better version of life. I I thought about it. Uh many people think that’s the answer to fmy paradox. There is also now a group of people looking at what they call a warfy successor. Basically, they kind of say, “Yep, we’re going to build super intelligence. Yep, we can’t control it.” So, what properties would you like to see in those systems? How important is it that it likes art and poetry and spreads it through the universe? And to me, it’s like I don’t want to give up yet. I’m not ready to decide if killers of my family and everyone will like poetry. I want to We’re still here. We’re still making decisions. Let’s figure out what we can do. Well, poetry is only relevant to us because poetry is difficult to create and it resonates with us. Poetry doesn’t mean jack shed to a flower. It’s more global to me. I don’t care what happens after I’m dead, my family is dead, all the humans are dead. Whether they like poetry or not is irrelevant to me, right? But but the point is like the things that we put meaning in, they it’s only us. The you know, a super massive black hole doesn’t give a about a great song. And they talk about some super value, super culture, super thing, super intelligence would like and it’s important that they are conscious and experienced all that greatness in the universe. But I would think that they would look at us the same way we look at chimpanzees. We would we would say, “Yeah, they’re great, but don’t give them guns. Yeah, they’re great, but don’t let them have airplanes. Don’t let them make global geopolitical decisions. So there are many reasons why they can decide that we’re dangerous. We may create competing AI. We may decide we’re going to shut them off. So for many reasons they would try to restrict our abilities, restrict our capabilities. For sure. Yeah. And there’s no reason why they would not limit our freedoms. If there is something only a human can do and I don’t think there is anything like that. But let’s say we are conscious, we have internal experiences and they can never get it. I don’t believe it. But let’s say it was true and for some reason they wanted to have that capability. They would need us and give us enough freedom to experience the universe, to collect those qualia, to kind of engage with what is fun about being a living human being, what makes it meaningful, right? But that’s such an egotistical perspective, right? That we’re so unique that even super intelligence would say, “Wow, I wish I was human.” Humans have this unique quality of confusion and creativity. There is no value in it. Mostly because we can’t even test for it. I have no idea if you’re actually conscious or not. So, how valuable can it be if I can’t even detect it. Only you know what ice cream tastes like to you? Okay, that’s great. Sell it now. Make a product out of it. Right. And there’s obviously variables because there’s things that people like that I think are gross. Absolutely. So, really, you can come up with some agent which likes anything or find any finds anything fun. God, why are you freaking me out right away? That’s the problem. This podcast is 18 minutes old and I’m like, we could just stop right now. Couple hours at least and then I fly here. I don’t Well, I don’t want to end. I have so many questions. But it’s just the problem is we got off to it. We just cut to the chase right away. And the the chase seems to be something that must be confronted because it is it’s right there. That’s it. That’s the whole thing. And I I’ve tried so hard to listen to these people that don’t think that it’s a problem and listen to these people that think that it’s going to be a net positive for humanity and oh God’s good. I feel better now. But it it doesn’t work. It doesn’t resonate. I wish they were right. Every time I have a debate with someone like that, I’m like, please come up with better arguments. Prove me wrong. I don’t want to be right on this one. I want you to show all the mistakes in my papers. I want you to show me how to control super intelligence and give us utopia, solve cancer, give us free stuff. That’s great, right? When you think about the future of the world and you think about these incredible technologies scaling upwards and exponentially increasing in their capability, what what what do you see like what do you think is going to happen? So there are many reasons to think they may uh cancel us for whatever reasons. We started talking about some game theoretical reasons for it. If we are successful at controlling them, I can come up with some ways to provide sort of partial solution to the value alignment problem. It’s very hard to value align 8 billion people, all the animals, uh you know, everyone because we disagree. We like many different things. So we have advanced virtual reality technology. We can technically give every person their own virtual universe where you decide what you want to be. You’re a king, you’re a slave, whatever it is you enter and you can share with others. You can visit their universes. All we have to do is figure out how to control the substrate, the super intelligence running those virtual universes. And if we manage to do that, at least part of a value alignment problem, which is super difficult, how do you get different preferences, multi-objective optimization essentially? How do you get different objectives to all agree? But when you think about how it plays out when when when like if you’re alone at night and you’re worried, what do you see? What do you see happening? So there are multiple levels of risk. Immediate is what we call eeky guy risk. I risk uh we lose meaning. You lost your job. You’re no longer the best interviewer in the world. Like what’s left? What are you going to do? Maybe some people will find some other uh kind of artificial things to do but for most people their job is their definition who they are what makes a difference to them for quite a few people especially in professional circles. So losing that meaning will have terrible impact in society. We always talk about unconditional basic income. We never talk about unconditional basic meaning. What are you doing with your life if basic needs are provided for you? Next level is existential risk. The concern is it will kill everyone. But there is also suffering risks. For whatever reason, it’s not even killing us. It’s keeping us around forever. And we would rather be dead. It’s so bad. What What do you see when you think of that? It’s hard to be specific about what it can do and what specific ways of torture it can come up with and why. Again, if we’re looking at worst case scenarios, I um I found this uh set of papers about what happens when young children have epileptic seizures, really bad ones. And what sometimes helps is to remove half of your brain. Oh jeez. Just cut it out. And there are two types of surgeries for doing that. One is to remove it completely and one is to kind of dissect connections leading to that half and leave it inside. So it’s like solitary confinement with zero input output forever. And there are equivalents for digital forms and things like that. And you worry that AI would do that to the human race. It is a possibility essentially new to us. Well, loss of control is a part of it, but you can lose control and be quite happy. You can be like an animal in a very cool zoo enjoying yourself, engaging in hiddenistic pleasures, sex, food, whatever. You’re not in control, but you’re safe. So, those are separate problems. And then there is for whatever reason, I don’t know if it’s malevolent payload from some psychopaths. Again, that would assume that they could control AI. I don’t think they will. But if they manage to do it, they can really put any type of payload into it. So think about all the doomsday calls, psychopaths, anyone providing their set of goals into the system. But aren’t those human characteristics? I mean those are characteristics that I think if I had to guess those those exist because in the future there was some sort of a natural selection benefit to being a psychopath in the days of tribal warfare that you if you were the type of person that could sneak into a tribe in the middle of the night and slaughter innocent women and children your genes would pass on. there was a benefit to that right so if it’s a human providing payload that’s what would show up if it’s AI on its own deciding what’s going to happen I cannot predict I’m just looking at worst case scenarios there are also game theoretic reasons where people talk about retrocausality where if right now you know what is that word uh like trying to influence the past uh say it again say retrocausality retrocausality causality causes oh okay so think about like weird time travel effects. Right now, if you’re not helping to create super intelligence, once it comes into existence, it will punish you really hard for it. And the punishment needs to be so bad that you start to help just to avoid that. My thought about it was that it would just completely render us benign, that it wouldn’t be fearful of us if we had no control, that it would just sort of let us exist. and it would be the dominant force on the planet and then it would stop if if human beings have no control over you know all the the different things that we have control over now like international politics um control over communication if if we have none of that anymore and we’re reduced to a subsistence lifestyle then we would be no threat it is a possibility I cannot say this will not happen for sure. But look at uh our relationship with animals where we don’t care about them. So ants, if you decide to build a house and there is an ant colony on that property, you genocide them, you take them out, not because you hate ants, but because you just need that real estate and it could be very similar. I again I cannot predict what it can do but if it needs to turn the planet into fuel raise temperature of a planet cool it down for servers whatever it needs to do it wouldn’t be concerned about your well-being it wouldn’t be concerned about any life right because it doesn’t need biological life in order to function as long as it has access to power and assuming that it is far more intelligent than us there’s abundant power in the universe there’s abundant power just the the ability to harness harness solar would be an an infinite resource and it would be completely free of being dependent upon any of the things that we utilize. And again, we’re kind of thinking what we would use for power. If it’s smarter than us, if it does novel research in physics, it can come up with completely novel ways of harnessing energy, getting energy. So, I have no idea what side effects that would have for climate. Right. Right. Why would it care about biological life at all? We don’t know how to program it to care about us. And even if we did it, if we if it felt like that was an issue, it felt like that if that was a conflicting issue, it would just change its programming. So usually then we start training AI, we train it on human data and it becomes really good very quickly, becomes superhuman and then the next level is usually zero knowledge where it goes all your human data is biased. Let me figure it out from scratch. I’ll do my own experiments. I’ll do some selfplay. I’ll learn how to do it better without you.