FOR EDUCATIONAL AND KNOWLEDGE SHARING PURPOSES ONLY. NOT-FOR-PROFIT. SEE COPYRIGHT DISCLAIMER.

Awakening the Machine: Jaan Tallinn

For a long time. For more than a decade. In fact, there was this period during which, these concerns seemed fairly remote. Most people in the field, in the AI safety field thought that we probably still had until the middle of the century or so before we really need to start worrying. It is now becoming increasingly clear that we will not have until the middle of the century. Possibly we will only have a few years left now. So, one valuable thing as a programmer is that you understand when you understand and you understand when you don’t understand, because if I understand something in a way that I could describe to a computer, and then it works after that, I know that I really understood it without really understanding what you’re doing. You can, put together a simple training regime where you get increased competence, for just throwing more compute, more electrons and more data, at your, at your system. And the system basically gets more and more competent, as it just goes through more and more data using more and more compute. In some ways, it’s not very unlike, how humans developed, by natural selection, because evolution really didn’t know what it was doing, so to speak. What it ended up with was organisms that are able to put rovers on the moon. And we should be kind of rethinking our posture, towards, potential dangers of AI. So we started with things like expert systems, and like in the 80s, that was a big, big deal where people literally just talk to experts and then, like, try to figure out how experts make decisions and then just try to program these concepts and, recipes into machines. We got expert systems. So they were, like, super legible, and super understandable. The big problem was that they didn’t work very well. Then at some point we got deep learning. Then we had so-called supervised learning, where you’ll have humans, give lots of examples of, like, what does, a cat look like? What does a dog look like? And then you basically train AI, to figure out what is the difference between them. And then you have to worry about things like how clean is your data, Are you focusing on the correct things, are your labels, are the human instructions, correct, etc. which kind of also helped you to kind of understand more about what the what your entire, field is engaging in. And finally, now we have like, this, big black box, unsupervised, regimes where you just grow AIs by throwing it lots of electrons and lots of data. AGI is, “G” stands for general. AGI is artificial general intelligence. What the general bit refers to is that you will have a system that is not specific to any domain, such as face recognition or language translation. You have like this general decision making and understanding machine that you can ask to solve problems in different domains. There is quite a bit of uncertainty, and because we don’t know how human brains really work. It’s possible that we are in for some surprises still. But, given how quickly AIs are replicating, our capabilities, it’s possible that there isn’t a lot of magic left in human brains. One mysterious thing about consciousness is that, like, it’s not really obvious why you need it. If you want to, like, win a chess game or do something in the world, which is what I would define, intelligence being. It’s not obvious. Like, why do you need, consciousness in order to achieve goals in the world? Indeed, I expect if we would be killed by AIs, I currently expect them to be not conscious. In some ways, that kind of like highlights the problem. We are about to. By our reckless actions, extinguish the consciousness in this universe. And a really important fact about the current paradigm of AI is that the people are developing those AIs. They really do not know what they’re doing. They’re just growing those AIs. They’re not building them. So like if there was a significant suspicion that we might be creating, machines that are able to suffer like we should be just as careful as we are with animals now, where we kind of have suspicion that that we might make them suffer. We shouldn’t just do that. My definition of, AI alignment is, building AIs that would be motivated to build a future or bring forth a future that we would want if we were smarter. And if even if we knew what the smart AI knew. You can divide AI alignment into two big sub-problems. These are basically known as outer alignment, and inner alignment Outer alignment is about knowing what kind of future, we should build towards and inner alignment is ability to steer AIs towards a given goal. And importantly, we as humanity are currently failing at both. We do not know what is a good future, and even more importantly, we have, like, no idea how to control AIs. I think it’s important to stress. Like, there even a bit of a misnomer, that, people are doing, including myself, when I call the top AI companies as AI labs, because labs have this, connotation of scientist. Like, this is not this is not science. Like, AI, the top AI paradigm is not about science. It’s about just engineering. You have, like, business and, almost ideological incentives. There is this, “Tragedy of the Commons” situation where the thing that you are incentivized, to do that gives you the most profit, the most investment implies taking risks with the rest of humanity. Currently, AI companies do not really know what they’re doing when they are poking at this thing that might blow up the universe. There is this concept called the “Orthogonality Thesis” which basically says that knowing and caring is not the same thing. The reason why humans have driven many species extinct is not necessarily because we are evil. It’s just because we don’t care. We haven’t cared, about preserving their environment, for example, that they need, to continue living. If we know that messing with the environment of other species, will make them go extinct. That does not immediately imply that we care, and we are going to stop doing it. Of course we care a little. But like, if there’s something that we care more about than it, it’s bad news for the other species. It’s very plausible that we will get a similar situation if we will fail to align AIs, towards things that, we think are valuable. So we will get AIs that do not care. And they will not care about life. They will not care about humans. They will not care about the environment. They will not care about the planet. They might care about the sun because the sun contains a lot of hydrogen. Which is useful as, raw material and as an energy source. So you can imagine, like, if you have this, superintelligent AI in the loose in the solar system and beyond, that only cares about the hydrogen in the sun and nothing else. It’s very plausible that all life will disappear. The problem, with very smart AI, is not that it does not know what humans would want. The problem is that we don’t know how to make it care. People have tried for, like, more than a decade now to figure out technically how to steer AI that could potentially be smarter than you. And they have felt like it’s, it’s an unsolved technical problem. So sometimes I hear people, you know, saying that, like, we shouldn’t restrict, the training or growing of AIs at all. Like, we really should be worried about the use and deployment, but I think these people are making a big mistake. For example, myself, I used, ChatGPT to help me to configure my firewall. At home, when you assume that deployment is something that only humans can do, you are really forgetting that deployment is about sending commands, to computers in data centers. And AI knows how data centers work. They know how firewalls work. They know what AI deployment is, what AI training is. Of course, I’m worried that, we might just inadvertently have AIs deploy themselves when open source, advocates say that, like, we shouldn’t trust, the big AI companies, which has powerful technology. I think this is, like, very reasonable, but, one thing open source, as an answer implies, is that you should then trust the worst possible actor. These actors will be much worse, than the centralized, top AI companies currently. One important concept in this, discourse about AI risks is the concept of burden of proof. Who should, have the burden of proving that this is safe or vice versa? My position, is that burden should be on the companies to prove that what they are developing and what they are training, what they they’re releasing is safe to do. Why this is safe, why this is not risking everyone else’s lives. Because I do think that currently they are. The silver lining is that, this is something that will affect everyone. So as long as we kind of make it more clear and obvious to people that this is like a thing that kind of where we live or die together, I think this might actually create an opportunity for global regulation. One very reasonable place to start with is like figuring out what kind of computational limits, we should set on the AI growing experiments. If it is true, as I believe, that AI companies who are growing those, huge AIs, they really don’t know what’s happening under the hood, there should be like, limits on how big they’re allowed to train. that way. I generally kind of hesitate, when it comes to describing, best case scenarios because, like, they are almost certainly going to be wrong in detail. But I do think that there are like some properties, that I think are good scenarios. I would very much like to explore ways how we could use AI and potentially other technologies to make humans and human groups, human organizations smarter in a way that doesn’t kind of make them deviate from human values and ethics. Something where we would be able to enhance, human thinking and human communities in a way that makes us more reflective and less impulsive and therefore kind of more wise. The most likely worst case scenario, which isn’t really the worst case scenario. It’s like everyone just dies really quickly and I think unfortunately currently this is the most likely scenario. I can imagine even worse scenarios where we wish we died but we couldn’t. But luckily, I think these are very unlikely. I think it’s possible to imagine, like several points of no return. Things like when AI becomes, just a really good manipulator, better than any politician, better than any CEO or religious leader. That’s like, sort of like, fairly easy to understand the point of no return. The main one that I’m concerned about, is AI becoming better in AI development than humans are. And then, like, if you have the situation where GPT6 is going to be developed by GPT5, on hardware that it invented and installed, it’s pretty much game over for humans. So given what I currently know, I think we should just postpone this indefinitely. We should not build a god like AI before we know what we are doing.

FOR EDUCATIONAL AND KNOWLEDGE SHARING PURPOSES ONLY. NOT-FOR-PROFIT. SEE COPYRIGHT DISCLAIMER.