The A.I. researcher Daniel Kokotajlo returns to the show to discuss a new set of predictions for how artificial intelligence could transform the world in just the next few years and how we avoid the most dystopian outcomes. Guest: Daniel Kokotajlo, executive director of the AI Futures Project Additional Reading: A.I. 2027 https://ai-2027.com/
well Casey today we’re going to talk about a forecast and that’s separate from a forkcast which is something different Yeah that’s uh what we call our end of the year predictions episode isn’t it I think so Uh but today we’re talking about something different which is this new report called AI 2027 Uh this is a report that I uh wrote about last week um and that has gotten a lot of attention in AI circles and policy circles this week It was produced by the AI Futures Project a Berkeleybased nonprofit led by Daniel Cocatello who listeners of the show may remember was a former Open AI employee who left the company last year uh and became something of a whistleblower warning about their uh reckless culture as he called it and is now uh spending his time trying to predict the future of AI Yeah And of course lots of people are trying to predict the future of AI But what gives Daniel a lot of credibility here is that in 2021 he tried to predict what he thinks would look like about now And he just got a lot of things right And so when Daniel said “Hey I’m putting together a new report on what I think AI is going to look like in 2027.” A lot of close AI observers said “Oh this is really something to read.” Yeah And Daniel didn’t just do this alone He also partnered with a guy named Eli Lifeland who is a an AI researcher and a very accomplished forecaster He’s won some forecasting competitions in the past And uh the two of them along with the rest of their group and Scott Alexander who writes the very popular Astrocodex 10 blog put together this very detailed what they call a scenario forecast Essentially it’s a a big report a website uh it’s got some you know sort of research backing it up and it is basically represents their best attempt to kind of synthesize everything they think is likely to happen in AI over the next few years into a readable narrative Yeah And if that sounds a little dull to you I’m telling you you should just go check this thing out It’s at a2027.com and it’s just super readable and it blows through stuff that feels very familiar right now like just sort of basic extrapolating from where we are today into getting to you know 6 months a year from now the world starts to look very very different and there is a lot of research that they have to support why they think that is plausible Yeah And I can imagine people reading this report or listening to us talking about it and say well that sounds like science fiction to me and we should be clear it is science fiction This is a fictionalized narrative that they have put together but I would say it is also grounded in a lot of empirical uh predictions that can be tested uh and confirmed or or you know verified It’s also true that some science fiction ends up becoming reality right If you look at uh movies about AI from past decades a lot of the things in those movies did end up actually being built So I think this um this report while it may not be 100% accurate is is at least represents a very rigorous and methodical attempt to sketch out what the future of AI might look like And here’s my bet If you put this conversation into a time capsule and revisited it in 2 years in 2027 my guess is we’re going to find that a good number of things in that scenario actually did come true And I hope we’re still doing a podcast in 2 years That’d be good Yeah that’d be great So my forecast is that this is going to be a good conversation Let’s bring in Daniel Cocatello Daniel Cocatello welcome back to Hardfork Thank you Happy to be here So you have just led this group that put together this giant scenario forecast AI 2027 What was your goal So our goal was to predict the future using the medium of a concrete scenario Um there is a small but exciting literature of attempts to predict the future of AI that use other methods which is also very important things like you know defining a capabilities milestone like here’s my definition of AGI here is my forecast for how long we’ll have until AGI based on these reasons and stuff And that’s great and we’ve done that stuff before We did a lot of that in the runup to this scenario Uh but we thought it would be helpful to have a actual concrete story that you can read Part of the reason why we think this is important is that it forces you to like think about everything and integrate it all into a coherent picture Well well I want to ask you a bit more about that So I mean the first thing I want to say about AI 2027 is it’s an extremely entertaining read Like it is as entertaining as most of the sci-fi that I have read by the end of it you get into scenarios where you know humanity’s survival is threatened And so whether you think it’s true or false it is like really engaging to read But my understanding of your aim here is that there is something practical about what you were trying to do right Can you tell us about sort of the practical idea of going through this exercise Yeah well I mean important background context The CEOs of OpenAI Anthropic and Google Demine have all publicly stated that they’re building AGI and that even that they’re building super intelligence and that they uh think that they can succeed by the end of this decade and that’s a really big deal and everyone needs to be paying attention to that Like I think a lot of people dismiss that as hype It’s a reasonable reaction to say like oh they’re just hyping their product but it’s not just the CEOs saying this It’s also the actual researchers at the companies And it’s not just people at the companies It’s also various independent people in academia and so forth And then also like you don’t just have to trust people’s word for it If you actually look at the evidence it really does seem strikingly plausible that this could happen by the end of this decade And then if it does happen then things are going to go crazy in some way or other We like it’s hard to predict exactly how but obviously if we do get super intelligent AGI uh what happens next is going to look like sci-fi right It will be like it’ll be it’ll be straight out of a sci-fi book except that it will be actually happening You you mentioned that uh if what the CEOs of tech companies say comes true we will be living in a sci-fi world And I think for a lot of people they’re content to sort of stop thinking there right they they might be willing to admit okay yeah if you invent super intelligence things will probably be crazy but like I’ll cross that bridge when we come to it You’re sort of taking a different approach and saying like no you’re going to want to start thinking right now about what it would be like if some of these claims start to come true So maybe we could get into what some of those claims are Sketch out for us what you think is very likely to happen just within the next couple of years Well I wouldn’t say very likely I should express my uncertainty Right So past discussion often focuses on a single milestone like artificial general intelligence or super intelligence We broke it down into a couple different milestones which we call uh superhuman coders superhuman AI researchers super intelligent AI researchers and then broad super intelligence Um and uh so we sort of like make our predictions for each of these stages Even the very first one I’m only like 50% confident that it’ll happen by the end of 2027 So I I a 50% chance that 2027 will end and there still won’t be any autonomous superhuman coding agents Um uh so I am uncertain you know but but uh let’s coin flipping we might also be living in a world where yes you do have an yeah exactly so 50% chance we do have autonomous fully autonomous artificial intelligences that can basically do the job of the cracked engineers by 2027 and then you okay well what’s the next milestone after that after that comes automating the full AI research process instead of just the coding because AI research is more than just coding and how long does it take to get to that well we have our guesses and in our scenario it happens like six months later you know yeah so in our story get the superhuman coders use them to go even faster to get to the superhuman AI researchers that are able to do the whole loop that really kicks things off and now you’re going much faster how much faster we say 25 times faster for the algorithmic progress at least of course your compute scale up is not going any faster at all because you still have the same amount of compute but you’re able to do the the sort of uh algorithmic progress 20 times faster 25 times faster uh then you start getting to the superhuman regime so you start getting systems that are It’s like qualitatively superior to the best humans at stuff And they’re also probably discovering new paradigms So we depict them going through multiple paradigm shifts over the course of the second half of 2027 ending up with something that’s just vastly superior to humans uh in every dimension uh by the end Yeah Let let me just sort of pause and and maybe underline a couple of things there I think you know most people might not understand why the big AI labs are obsessed with automating coding right Most people are not software engineers So they kind of don’t care how much of it is automated But by the time you get to software that is mostly writing itself it unlocks this other world of possibilities And you just sort of sketch out a vision where once we get to a point where the uh sort of AI coding systems are better than almost every human engineer or maybe every human engineer then this other thing becomes possible which is now you can just set this thing to work trying to figure out how to build AI itself Right Is that what is that what I’m hearing you say Basically I’d break it down into two stages So So I think the coding is is separate from the complete automation as I previously mentioned I think that uh I expect to see systems that are able to do all the coding extremely well but might lack research taste for example They might lack good judgment about what types of experiments to run And so that’s why they can’t completely automate the research process And then you have to make a new system or continually train the old system so that it gets that taste it gets that judgment Similarly they might lack coordination ability they might be uh not so good at working together in large organizations of thousands of copies at least initially but then you fix that and you come up with new methods and you do additional training environments and get them good at that sort of thing and that’s what we depict happening over the first half of 2027 and we depict it happening in only half a year because it goes faster because they’ve got all the coding down pat and so even though humans are still directing the whole process they just give orders to the coding agents and they quickly make everything actually work um And then you know halfway through the year they’ve succeeded in making new training runs that train the skills that they that the AIs were missing So now they’re not just coding agents They are able to do uh the research taste as well They’re able to come up with the new ideas They’re able to come up with hypotheses and test them And they’re able to work together in big sort of like hive mind clusters of thousands and thousands of them And that’s when things really kick off You know that that’s when it really starts to accelerate In your scenario you have this sort of choose your own adventure ending um where after this thing you call the intelligence explosion where the superhuman AI coders get into AI R&D and they start automating the process of building better and better AIs you sort of have two buttons that you can click and one of them sort of unspools the the good place ending where we you know decide to slow down AI development and really get these things under control and solve alignment and then the red button you push that and it goes into this very dark dystopian scenario where we lose control of AI they start deceiving and scheming against us and ultimately maybe we all die um why did you decide to give people the the option of choosing one of those two endings rather than just sketching what you believe to be the most probable outcome So we did start by sketching what we believe to be the most probable outcome and it’s the uh the race ending the one that ends with the misand uh and then we were like well this is kind of depressing and sad and there’s a whole bunch of stuff that we didn’t get to talk about because of that Um and so we wanted to then have a different ending that ended differently In fact we wanted to have like a whole spread of different possible uh outcomes but we were limited by time and labor and we were only able to pull together one other outcome which is the one that we depicted in in the slowdown ending So in the slowdown ending uh they solve the alignment issues and they they actually get AIS that are uh actually you know what they say on their tin They’re not faking it they’re they just actually have the goals and values that were put into them or that that the company was trying to train into them You know it takes them a couple months to like sort that out That’s why it’s a slowdown They had to like pivot a lot of their compute and energy towards figuring that stuff out Um but they succeed and so then in that ending uh we still have this crazy arms race with China and we still have this crazy geopolitical crisis Um and in fact it still ends in a similar sort of way with this massive arms buildup on both sides this massive integration into the economy and then ultimately a peace treaty I’m curious Daniel if the events of the last week in Washington uh the tariffs this looming trade war with China um have affected your forecast at all I mean we’ve been we’ve been it iteratively improving it but like the core structure of it was basically done a few months ago so this is all new to us and wasn’t really part of the forecast How would it change things Well if the trade war continues and causes a recession and stuff like that it might uh just generally slow the pace of AI progress but not by much I think like say it makes compute 30% more expensive so that the companies are able to buy 30% less of it Um maybe that would translate to like a 15% reduction in overall research velocity over the next few years which would mean that the milestones that we talk about happen like a few months later instead of when they do So the story would still be basically the same So one of the things I think is most interesting about your project is the bets and bounties section where you are going to pay people for finding errors in your work for convincing you to change your mind on key points or for drafting some alternate scenarios So talk to me a little bit about how that became part of this project So like you know I I come from the sort of rationalist community background which is big into making predictions and making bets putting your money where your mouth is So I have a sort of aesthetic interest in doing that sort of thing But then also specifically one of the goals of this project is to get people to think more about this stuff and to you know do more scenario forecasting along the lines of what we’ve done We’re really hoping that people will counter this with their own reasonably detailed you know alternative pathways that represents their vision of what’s coming Um and so we’re going to give out a few thousand dollars of prizes uh to to try to mildly incentivize that Um and then as for the bounties thing already we’ve gotten dozens of people being like you say this but like isn’t this a typo or like you you know this this feels wrong and so I have a backlog of things to process but I’m going to get through it I’m going to like you know pay out the little the little payments and fix all the the little bugs and and stuff like that Um and uh I’m just quite heartwormed to see that that level of engagement And have you uh like taken any uh bets on like different scenarios so far I think so far I’ve done like one or two Uh but mostly there’s just a backlog I need to work through Now Daniel you said you’ve been getting some good responses from people at the AI companies to this scenario forecast Um I did a bunch of calling around when I was writing about this and after we spoke I talked to a bunch of different people both in the AI research community and outside of it And I would say the most frequent reaction I got was just kind of uh disbelief Um one uh person I talked to a a prominent AI researcher said he thought it was an April Fool’s joke when I first uh showed him this scenario because it just sounded so outlandish You know you’ve got Chinese espionage and the models going rogue and the superhuman coders and like it all just seemed fantastical and it was almost like it they didn’t even think it was worth engaging with because it was so far out I’m curious if you’ve gotten much of that kind of reaction and what your response is Uh well you know go write your own damn scenario then I would say you either will write a scenario that doesn’t seem outlandish which I will completely tear apart as unrealistic and just assuming basically that AI progress hits a wall or you’ll write a scenario that does feel very outlandish but perhaps in different ways than ours do Again like are they actually going to get to AGI and super intelligence by the end of this decade If so you can’t possibly write that in a way that’s not outlandish Uh it’s just a question of like which outlandish thing are you going to write And if you think maybe this is not going to happen and it’s going to hit a wall yeah that’s possible too I think that’s reasonable Um I don’t think it’s the most likely outcome Like I I do actually think that probably by the end of this decade we’re going to have super intelligence and well and and say more about that because you know I I assume that a lot of our listeners like think either truly think that it will hit a wall or they’re just sort of counting on it hitting a wall so as not to have to reckon with any of the scenarios that you describe So like what is your message to the person that’s just like it’ll probably hit a wall I mean I don’t know read the literature Like there there’s these people are not going to read the literature They listen to podcasts specifically so they don’t have to read the literature Yeah fair Well um I could point to specific parts of the literature like like benchmarks for example and the trends on them Um so I would say the benchmarks used to be terrible but they’re actually becoming a lot better uh meter in particular has these uh agentic coding benchmarks where they actually give AI systems access to some GPUs and say have fun you have like eight hours to make progress on this research problem um good luck and then they measure how good they are compared to human researchers given the same setup and um you know line goes up on the graph it seems like in a year or two they’ll have AIs that are able to just autonomously do eight hour long ML research tasks of a wide variety of such tasks um you know uh on these sorts of things and and that’s not AGI that’s not super intelligence but that is maybe the first milestone that I was talking about superhuman coder right so so I point to those sorts of trends and then separately I would also to just do the appeal to authority Like if you’re not going to read the literature if you’re not going to look at the if you’re not going to sort of form your own opinion about this and you’re still just deferring to what other people think well then I will say yeah there’s a bunch of naysayers out there who are saying this is all never going to happen It’s just fantasy But also there’s a bunch of extremely credible people with amazing track records uh both inside the companies and outside the companies who are in fact taking this extremely seriously Yeah I also want to read including our scenario like like you know Yashu Benjio for example read an early draft of our thing and liked it and gave us some feedback on it and then we we put a quote from him at the top saying everyone should read this it’s plausible He’s a he’s a pioneering AI researcher Another genre of criticism I’ve heard of this forecast is uh from people who are just questioning the idea that if you get AIs that are superhuman at coding they will kind of be able to bootstrap their way to general intelligence And I I just want to read you a a quote from an email that I got from uh David Aur who is a a very well-known economist at MIT And I had asked him to look at the scenario and sort of react to it and uh with a particular eye on like what might this be missing as far as how it sort of assumes this this sort of easy and fast jump from superhuman coding to something like AGI And I’ll just read you what he said He said “LMs and their ilk are superpowered incarnations of one incredibly important and powerful part of our cognition The reason I say we’re not on a glide path to AGI is that simply taking this capability to 11 does not substitute for the parts that are still missing I think that humanity will get to AGI eventually I’m not a dualist I just don’t believe that swimming faster and faster allows you to fly What is your reaction to that I agree Uh we depict this in the course of the story So if you if you read AI 2027 um they have something that’s like LLMs but with a lot more reinforcement learning to do long horizon tasks and that is what counts as the first superhuman coder Um so it’s already somewhat different from the systems of today but it’s still broadly similar It’s still sort of maybe the same fundamental architecture just a lot more training a lot more scaling up and in particular a lot more training specifically on long horizon agentic coding tasks Um but that’s not itself AGI I agree That’s just the superhuman coder that you get early on And then you have to go through several more like paradigm shifts to get to actual super intelligence And we depict that happening over the course of 2027 So a key thing that I think that everyone needs to be thinking about is uh this this takeoff speeds variable um how much faster does the research go when you’ve reached the first milestone and how much faster does the research go when you reach the second milestone and so forth And we are of course uncertain about this like we are about many things We say in the scenario that we could easily imagine it being five times slower than we depict and taking sort of like 5 years instead of one year Uh but also we could imagine it being five times faster than we depict and taking like two months you know Um so we want to do a lot more research on that Obviously we you can if you want to know where our numbers are coming from uh go to the website There’s a a a tab that you can click on that lists has a bunch of sort of like back of the envelope calculations and little mini essays where we like generated the quantitative estimates that that are the skeleton of the story One other piece of criticism I’ve seen of this project that I wanted to ask you about was from a researcher at anthropic named Saffron Hang who argued on X that she thought that your approach in AI 2027 was highly counterproductive basically that you were uh in danger of creating a self-fulfilling prophecy uh by making these sort of scary outcomes uh very legible by sort of uh you know burying some assumptions that you were essentially making the bad scenario that you’re worried about more likely to actually happen What do you make of that I’m quite worried about that as well and this is something we’ve been like fretting about since day one of the project First of all there is a long history of this sort of thing seeming to happen in the field of artificial general intelligence research Uh most notably um Ellie Ziowski who is the sort of like I don’t know father of like worrying about AGI at least in this generation people you know Alan Turring also worried about it but like anyhow um Sam Alman specifically tweeted you remember this tweet Yeah Sam specifically said like hats off to the owski for like raising awareness about AGI It’s happening much faster now because of his doomsaying because it’s caused a bunch of people to like pay more attention to the possibility and to like you know start investing in these companies and so forth So I was sort of like a I don’t know twisting the knife at him because he obviously doesn’t want this to happen faster He thinks we need more time to prepare and make it safe and so forth But um it does seem like there’s been this effect where people talking about how powerful and scary AGI could be has maybe caused it to come a little bit faster and caused people to like wake up and and race harder towards it Um and similarly I’m worried about causing something like that with AI 2022 Like I one of the like subplots in AI 2027 is this whole like concentration of power issue of like who gets to control the army of super intelligences right And in in in the race ending it’s sort of a moot question because the army of super intelligence is is just pretending to be controlled and so is not actually listening to anyone when it counts Um but in the slowdown ending they do actually align the AIS and so they are actually going to do what they’re told and then who gets to say that right And the answer in our slowdown ending is the the oversight committee which is this like ad hoc group of people that is some CEOs and the president who get together and like share power over the army of super intelligences But uh what I would like to see is something more democratic than that Something where the power is more distributed Um I’m also afraid that it could be less democratic than that Like at least we get an oligarchy with this committee like it could very easily end up a dictatorship where one person has absolute control over the army of super intelligences Um this is yet another example of like how I’m trying to like not have the self-fulfilling prophecy happen Like I don’t want people to read this and be like I’m a CEO I can make a lot of money by building or like you know may maybe Yeah So so so but but all that being said Yeah to any of our evil villain uh listeners out there steepling your fingers in your uh in your in your uh lair under a mountain knock it off Yeah So so all that being said we are taking a gamble that uh like you know sunlight is the best disinfectant like the the best way forward is to just generally tell the world about what we think is coming and hope that even though many people will react to that in exactly the wrong ways enough people will react to that in the right ways that overall it will be good Um because I am tired of the alternative of like hush hush keep everything secret do backroom negotiations and hope that we get like the right people in the right rooms at the right time and that they make the right decisions I think that that is kind of doomed Um so I’m I’m sort of placing my faith in humanity and telling it as I see it and hoping that in so far as I’m correct people will wake up in time and you know overall that the outcome will be better Thank you Daniel Thanks Daniel Thanks guys