“AI is powerful. It is going to be extremely, unbelievable, unbelievably powerful, and it is because of this power that’s where the safety issues come up.” [Note: this is a post of an old interview, prior to Ilya leaving OpenAI]
Ilya Sutskever | AI will be omnipotent in the future | Everything is impossible becomes possible
TRANSCRIPT. all right welcome um there were lots of people who had lots of interesting questions so I gave myself some note cards so I’ll I’ll I’ll be prepared but um uh maybe we start with this um you have always been an deep learning maximalist um even very very early on what gave you the conviction to say look if you just push this to larger and larger models we’re going to to see really unexpected interesting Behavior what what gave you the conviction that early on so I claim that to get this conviction that to believe that large neural networks can do amazing things you need to have two beliefs one of the belief one of the beliefs is a little bit harder to get to the other one is easier so the easy belief is that the human brain is Big the human brain is big and the brain of a cat is smaller and the brain of an insect is smaller still and we correspondingly see that humans can do things which cats cannot do and so on that’s easy the hard part is to kind of say well maybe an artificial neuron the kinds the kind of neurons that we have in artificial neural networks is not that different from a biological neuron as far as the essential information processing is concerned so in other words of course the AR the biological neuron is very complicated and it does so many different things but when it comes down to it you have signals in Signal out maybe it’s a pretty not maybe you can explain a lot with a pretty simple artificial neuron and if you just allow yourself to say yeah yeah they’re different yeah yeah biological neurons are more complex but let’s just say suppose they are similar enough then you say yeah okay V now have an existence proof that large neural nets all of us can do all these amazing things so the existence is there can we then somehow make it so for that we need to be able to train but if you that’s the kind of chain of reasoning which you know in the environment of my you know when I was in graduate school with Jeff I think it was we were thinking about neural Nets it was perhaps more possible more feasible to make this realization than it would have been elsewhere yeah certainly we tried Neal Nets before and we didn’t quite get to the same results because we’re doing at a much smaller scale and so on um interesting where um let’s start with so what’s your definition of AGI how what’s your mental picture yeah so AGI so at open AI we have a document which we call the open a charter which outlines the goal of open Ai and there we offer a definition of AGI and we say that an AGI is a computer system which can automate the great majority of intellectual labor that’s one useful definition m in some sense an AGI would be the intuition there is it’s a computer that’s as smart as a person so you might for example have a coworker MH that’s a computer so that would be a def a definition of AGI which I think is intuitively satisfying the term is a bit ambiguous because AGI the g means general so is it generality that we want that we care about in the AGI but it’s actually a bit more than generality we care about generality and competence needs to be General in a sense that it can respond sensibly when you throw things at it but it needs to be comp competent so that you when it does something you ask it a question or ask it to do something it will do it yeah I like the sort of very practical definition at the end of the day because it gives you some measurement where you can can figure out how close are you do do you think we have all the ingredients to to get to AGI um if not what’s missing kind of in the stack it’s a complicated stack already um I trans per forers really all we need kind of paying homage to the famous U attention paper yeah you know I won’t be overly specific in my answer to this question but I will say that I think that no I’ll comment on the second part of the question is is is Transformers is all we need and I think that the question is a bit wrong because it implies something binary it implies Transformers are are either good enough or not good enough but I think it’s better to think about it in terms of tax where we have Transformers and they’re pretty good mhm maybe we could have something better that would be maybe more efficient or maybe you’ll be faster but we as we know when you make the Transformers large they still become better they might just become big might be becoming better more slowly so while I am totally sure that it will be possible to improve very significantly on the on the current architectures that we have even if we didn’t we would be able to go extremely far do you think it matters what the algorithm is so so for example an lstm versus a Transformer just scaled up sufficiently maybe that’s an efficiency Delta or something like that but don’t we end up in the same same place at the end so I would say almost entirely yes with a caveat so there are two caveats Lis so I’m just thinking of how what level of detail to go here you know maybe I will I will I will skip the detail how many people in the audience know what an lstm is Oh see it’s a around here so I think we’re mostly okay let’s dig let’s let’s dig in then so I would argue that with a few if we made a few simple modifications to the lstm their hidden states are quite small if you somehow made it larger and then we were to go through the trouble of figuring out how to train them cuz lstms are recurrent neural network works and we kind of forgot about them we haven’t put in the effort to cuz you know how neural training works you have the hyper parameters well how do you set them it’s like you don’t know how do you set your learning rates if it doesn’t learn can you explain why and so this kind of work has not been done for lstms so that’s why our ability to train them is more reduced but had we done that work so that we were able to train the lstms and we just did some simple things to increase their hidden State size I think they would be worse than Transformers but we would still be able to go extremely far with them also okay um how good is our understanding of scaling laws like if we if we scale these models up how confident are you in being able to predict capabilities of these particular models how good is that science so that’s a very good question the answer is so so I was hoping for a more definitive answer well for it so so is a very definitive answer it means we are not great but we are not absolutely terrible either but we are not great definitely not great so what the scaling LW tells you it uh relates it’s a relationship between the inputs that you put into the neural network and some kind of a simple to measure performance simple to evaluate performance measure like you your next word prediction accuracy M and that relationship is very strong but what is challenging is that we don’t really care about next word prediction we care about it indirectly we care about the other incidental benefits that we get out of it and our and so our so for example you all know that if you predict the next word accurately enough you get all kinds of interest in emerging properties those have been quite hard to predict or at least I’ll say I’m not aware of such work and if anyone is looking for interesting research work pro problems to work on that would be one I will say I will mention one example something that we’ve done at open AI in our in in our runup to GPT 4 where we tried to do a scaling law for a more interesting task which is predicting accuracy at solving coding problems we were able to do that accurately very accurately and that’s a pretty good thing because this is a more tangible metric it’s not it’s still it’s it’s an improvement over next step next word prediction accuracy as far as things that are relevant to us so in other words it’s more relevant to us to know what the coding accuracy is going to be ability to solve coding problems compared to just ability to predict and Export it still doesn’t answer the really important question of can you predict some emergent behavior that you haven’t seen before okay um speaking of these capabilities that are kind of emerging capabilities which one surprised you the most as these models scaled what what was the thing where you said like well I’m kind of astonished these models can do this it’s a very difficult question to answer because it’s too easy to get used to where things are so there definitely have been times when I was surprised but you adapt so fast it’s kind of crazy I think maybe the big surprise for me is you know it may it may sound a little odd probably to most people in this audience but the big surprise for me is that neural networks work at all because when I was starting my work in this area they didn’t work or it was like let’s define what it means to work at all it means they could do they could work a little bit but not really not in any serious way not in a way that anyone except for the most intense enthusiasts would care about and so now we see yeah like those neural Nets work so I guess the artificial neuron really is at least somewhat related to the biological neuron or at least that basic assumption has been validated to some degree what about like an emergent property was the one that sticks out to to you like for example I don’t know code generation or did you may maybe it was different in your mind maybe you you just once you saw like hey neural Nets can work and they can scale yeah of course all these sort of properties will emerge because you know at at the limit point we’re building a human brain and humans know how to code and humans know how to reason about tasks and so on um was that did you just expect all of that or did uh I’ve definitely been surprised and I’ll mention why because the human brain can do those things it’s true but does it follow that our training process will produce something similar so so it was definitely very amazing I think yeah seeing seeing the coding ability improved quickly that was quite quite a sight to be seen and for coding in particular because you know it went from no one has ever seen a computer code anything at all ever there was a little area of computer science called program synthesis mhm which maybe it was very Niche and it was very Niche because they couldn’t have any accomplishments it was a very they had a very difficult experience and then this neural came in and said oh yeah code synthesis like we’re going to do we’re going to accomplish what you hope were hoping to achieve one day like tomorrow so that was yeah deep learning just just out of curiosity when you write code how much of your code is yours how much of your code is I mean like collaboration but I I I do eny en jooy I do enjoy it when the neural net writes most of it all right let’s let’s switch TCT here a little bit um as these models get more and more powerful um it’s worthwhile to to also talk about AI safety and uh uh and open AI has has released the document just uh just recently that where you’re one of the unders signers um uh Sam has testified in front of Congress what what worries you most about AI safety yeah I can talk about that so let’s take a step back and talk about the state of the world so you know you’ve had the AI research happening and it was exciting and now you have the GPT models and now you all get to play with all the different chat bot and assistance and you know B and chat GPT and you say okay that’s pretty cool it can do things and indeed there already are you can start perhaps worrying about the implications of the tools that we have today and I think that it is a very valid thing to do but that’s not where I allocate my concern M the place where things get really tricky is when you imagine fast forward in some number of years a decade let’s say how powerful will a I be of course with this incredible future power of AI which I think will be difficult to imagine frankly with an AI this powerful you could do incredible amazing things that are perhaps even outside of our dreams like if you can really have a dramatically powerful AI but the place where things get challenging are directly connected to the power of the AI it is powerful it is going to be extremely unbelievable unbelievably powerful and it is because of this power that’s where the safety issues come up and I’ll mention three I I personally see three you know when when you get so you you alluded to the letter M that uh we posted at open AI a few days ago actually yesterday about what we about some ideas that we think would be good to implement to navigate the challenges of super intelligence now what is super intelligence why did we choose CH to use the term super intelligence the reason is that super intelligence is meant to convey something that’s not just like an AGI with AGI we said well you have something kind of like a person kind of like a coworker super intelligence is meant to convey something far more capable than that when you have such a capability it’s like can we even imagine how it will be but without question it’s going to be unbelievably powerful it could be used to solve incomprehensible hard problems if it is used well if we navigate the challenges that super intelligence POS poses we could we could radically improve the quality of life but the power of super intelligence is so vast so the concerns the concern number one has been expressed a lot and this is the scientific problem of alignment you might want to think of it from the as as an analog to nuclear safety you know build a nuclear reactor you want to get the energy you need to make sure that it won’t melt down even if there’s an earthquake and even if someone tries to I don’t know smash a truck into it y so this is the super intelligence safety and it must be address in order to contain the vast power of the super intelligence this called the alignment problem one of the suggestions that we had in our in the PST was an approach that an international organization could do to create various standards at this very high level of capability and I want to make this other point you know about the post and also about um R CEO Sam Alman Congressional testimony where he advocated for regulation of AI the intention is primarily to put rules and standards of various kinds on the very high level of capability you know you could maybe start looking at gp4 but that’s not really what is interesting what is relevant here but something which is vastly more powerful than that when you have a technology so powerful it becomes obvious that you need to do something about this power that’s the first concern the first challenge to overcome the Second Challenge to overcome is that of course we are people we are humans humans of interests and if you have super intelligence is controlled by people well who knows what’s going to happen I do hope that at this point we will have the super intelligence itself try to help us solve the challenge in world that it creates this is not no longer an unreasonable thing to say like if you imagine a super intelligence that indeed sees things more deeply than we do much more deeply to understand reality better than us we could use it to help us solve the challenges that it creates then there is the third challenge which is the challenge maybe of natural selection you know what the Buddhists say that change is the only constant so even if you do have your super intelligences in the world and they are all we managed to solve alignment we managed to solve no one wants to use them in very destructive ways we managed to create a life of unbelievable abundance which really like not just not just material abundance but Health longevity like all the things we don’t even try dreaming about because they’re so obviously impossible if you’ve got to this point then there is the third challenge of natural selection things change you know you know that natural selection applies to ideas to organizations and that’s a challenge as well maybe the neuralink solution of people becoming part AI will be one way we will choose to address this I don’t know but I would say that this kind of describes my concern and specifically just as the concerns are big if you manage man it is so worthwhile to overcome them because then we could create truly unbelievable lies lives for ourselves that are completely even unimaginable so it is it is like a challenge that’s really really worth overcoming