“The first thing to recognize is that the current incentives are wrong. I mean, we are in an arms race condition, both with respect to the individual companies that are doing this work in America and in the West… I think the real solution ultimately is to achieve a world that is politically sane enough that offers enough of a basis for cooperation at a global scale such that we can get out of this arms race condition. You know, we wanna live in a world where we don’t fear the Chinese and they don’t fear us.” — Sam Harris
“Show me the incentive and I’ll show you the outcome.” — Charlie Munger (1924 – 2023)
The solution to “God-like AI” – What would a totalitarian society that is not well disposed to our own do if it had God-like power? And we don’t wanna find out. That said, I think the real solution ultimately is to achieve a world that is politically sane enough that offers enough of a basis for cooperation at a global scale such that we can get out of this arms race condition. You know, we wanna live in a world where, you know, we don’t fear the Chinese and they don’t fear us. And we don’t live in a world where at least one of those parties is right to be feeling fear at this moment because the other party really is committed to goals that are inimical to everything that we value and are right to value. We should get powerful AI before those people. – [Narrator] How can we develop artificial intelligence responsibly? The risk of self-improving AI – Well, we can mean at least two things by artificial intelligence. There’s the narrow version where we have increasingly competent machines. I mean, now we’re experiencing large language models that are quite amazing in their ability to produce text. And then there’s the wider version, often called AGI, or artificial general intelligence, which refers to a human-like capacity to do many different types of things well, wherein there there’s no real specialization and there’s no degradation of function across those domains, right? So the better you get at parsing language, that doesn’t mean you get worse at solving math problems or recognizing faces. So the moment we get something that is truly general, that is human-like in its ability to solve problems across a range of environments and with no degradation and it’s learning, it just becomes, you know, in the end, self-improving, one thing becomes obvious, right? First, this thing will be immediately superhuman because we will not have built any of these individual capacities at a level lower than human, right? So, you know, your calculator in your phone is already superhuman for arithmetic. There’s no way the calculator we put into the AGI is gonna be worse than the one we put in your phone, right? So the moment we get this omnibus suite of capacities that are truly general, we have to recognize that that human level intelligence is a mirage that we never even, you know, arrived at for even a moment, right? We just crossed over from this piecemeal kinda superhuman narrow versions of intelligence. You have a superhuman calculator, and you now have a superhuman large language model. When all this gets knit together by whatever architecture, you will suddenly be in the presence of the most competent mind you’ve ever met, right? And I think that the thing that’s important to recognize there is that if we’re truly talking about general intelligence, we’re talking about autonomy, we’re talking about a relationship, therefore, we’re talking about being in the presence of another mind, whether it’s conscious or not, we can leave consciousness aside because I think it’s genuinely uncertain whether a consciousness comes along for the ride as you scale up in intelligence. I happen to think there’s no reason to expect that at this point. So whether the lights are on or not in our robots or in our most powerful computers, they’ll certainly seem to be conscious because we’ll build them that way, or certainly we’ll build some of them that way. And we might just lose sight of the problem as to whether it’s interesting intellectually or ethically to figure out whether these systems are conscious, because they’re just going to seem conscious. I mean, certainly if we build humanoid robots that are more intelligent than we are, we will feel that we’re in relationship to conscious entities, and many things follow from that. Two levels of risk So there are really two levels of risk here. There’s the risk that that bad people, or, you know, badly intentioned people or unwise people will do bad things with their AI, whether it’s, you know, narrow AI of increasing strength or the general AI that they build. And that’s one bad outcome that we can try to safeguard against, and that’s not easy. But there’s this additional problem and probably deeper problem that, in the presence of truly autonomous AGI, general intelligence that is super human, we need not merely worry about human bad actors. We need to worry about what’s called the alignment problem. You know, whether this now more competent mind is aligned with our interests or disposed to be, you know, realigned with our interests whenever we detect that there’s some daylight between what it’s doing and what we want, right? I mean, it is just the kind of mind that cares what we want. Have we built it in such a way that it could ever lose sight of what is good for us, right? And it’s not obvious in advance all the ways in which a mines more intelligent than our own could grow unaligned with our interests and depart from this ongoing effort to make the world better and better for us, right? I mean, just imagine by analogy what it’s like for every other species on Earth watching humans grow more and more powerful, right? Human culture, human society, human technology. You know, at one point we were hairless apes with sticks and rocks, you know, and flint tools. And that gave us this overwhelming advantage, right? Just our ability to cooperate through language and the most primitive technology we began to leverage, already there we were unstoppable when you consider the career of any other species, even our closest cousins, you know, the Neanderthals, which we very likely wiped out. What will it be like to be in the presence of minds that, again, whether they’re conscious or not, are so much more competent than we are and so busy doing things that they are forming instrumental goals that we can’t possibly understand, right? And they’re doing all of this so quickly, right? I mean, just speed on its own could be totally destabilizing. I mean, just imagine we built AI that was no smarter than the 10 smartest people together in a room, but it just worked a billion times faster, right? Well, what would it be like to be in relationship to 10 people who every time, you know, you stop to think for a second, they did 32 years, you know, a billion seconds worth of cognitive work, right? What would that conversation be like? I mean, it’s unimaginable, right? And that’s why this whole thing is described as a singularity, right, or an intelligence explosion. And the moment this really becomes a concern, once we imagine that the machines themselves could become recursively self-improving, that they could be the agents of improving their own software or they could build the next generation of more competent machines, right? The moment we engineer anything like that, and there’s no reason to think we won’t do that at this point, something like an intelligence explosion certainly becomes conceivable. And whether it happens slowly or quickly, again, I think the thing to recognize is that we have to understand that we will be in relationship to other minds more powerful than our own, right? And unless they have been built so as to have, at bottom, a core concern to more faithfully approximate what we want, you know, ad infinitum, it remains a genuine fear that we could build something that could, at a certain point, no longer care what we want. The AI arms race The first thing to recognize is that the current incentives are wrong. I mean, we are in an arms race condition, both with respect to the individual companies that are doing this work in America and in the West. And we’re in an arms race with any other society that is close to getting into the end zone themselves, and I think China being the most obvious example. When you look at what we should do globally, geopolitically, I think we need to win the arms race, right? So that doesn’t solve most of our problems with AI or much less all of them. But it solves the problem of what would a totalitarian society that is not well disposed to our own do if it had God-like power? And we don’t wanna find out, right? So we don’t want the Chinese to get the perfect self-improving AI before we do, or even the imperfect, but nonetheless powerful self-improving AI before we do. We don’t want them to get the lethal autonomous drones or the robot army before we do. And if my thinking has changed on anything here, and this is going in a very dystopian direction, but, you know, I used to think that we didn’t want to militarize and weaponize the most powerful AI as we acquired it. I thought that autonomous weapons was almost definitionally a bad thing. I don’t think I believe that anymore. I think it’s conceivable that we would build weapons that are autonomous, that are better than humans at making judgements with respect to, you know, when to use lethal force, which is to say their error rate will be acceptable to us, even though they’ll have one because it’ll simply just be better than keeping a monkey in the loop, in the same way that self-driving cars will almost certainly will exist and be better than we are. And at that point, we’ll consider it unethical to be driving our own cars because we’ll just, you know, be reliably killing each other because we’re just bad at it. So I think that’s conceivable, but more to the point, I think China is guaranteed to attempt to build this kind of weaponry. And what we need are weapons that can counter that. And we need to be more powerful. I just think we’re in an arms race that’s now unavoidable, analogous to the arms race we had with nuclear weapons. And, unfortunately, the game theory is such that, you know, it’s just, we can’t opt out. I think it would be bad for us to opt out. That’s not a future we wanna live in. That said, I think the real solution ultimately is to achieve a world that is politically sane enough that offers enough of a basis for cooperation at a global scale such that we can get out of this arms race condition. You know, we wanna live in a world where we don’t fear the Chinese and they don’t fear us. And we don’t live in a world where at least one of those parties is right to be feeling fear at this moment because the other party really is committed to goals that are inimical to everything that we value and are right to value, right? I mean, there are good actors and bad actors. It’s not that everyone has an equal claim upon the moral high ground here. You know, there’s certain human futures that we don’t want and we are right not to want them. And the fact that you can find millions, and in some cases even billions of people who want them, doesn’t mean they’re not wrong, right? We should get powerful AI before those people. I’ve changed my thinking to some degree on how much we should lean into the arms race insofar as it’s impossible to opt out of it. But ultimately we need to move on another track toward something like political sanity where we can step off this current course.