“Most are dead within hours, the few survivors like preppers in bunkers. This is why Sam Alman has a bunker. He’s a prepper. Somewhere in New Zealand, I think.” — Wes Roth (44:50)

“I try not to think about it too much, but I have guns, gold, potassium iodide, antibiotics, batteries, water, gas masks from the Israeli Defense Force, and a big patch of land in Big Sur I can fly to.” — Sam Altman (2016)

FOR EDUCATIONAL AND KNOWLEDGE SHARING PURPOSES ONLY. NOT-FOR-PROFIT. SEE COPYRIGHT DISCLAIMER.

Experts Warn: AI Takeover by 2027… Wes Roth

Learn More

so what happens to AI by 2027 some AI researchers are proposing a scenario and it’s not looking great you might remember the name Daniel Katalo he was the former OpenAI employee that was the whistleblower for OpenAI flagging potential artificial intelligence risks he could have lost a lot of money as OpenAI equity because OpenAI at the time had a clause saying that you can’t disparage the company or lose some of your vested equity some people were warning about a culture of risk and retaliation and calling for whistleblower protection and Scott Alexander was the person behind Slate Star Codex he was forced to shut it down or chose to shut it down in part it seems like because the New York Times threatened to reveal his actual name and eventually sounds like they did so he took down the Slate Star Codex and this is their prediction for what will happen over the next few years as AI develops as it improves as AI agents become better and better at a variety of different things but most importantly at improving AI at doing AI research they’ve recently appeared on a number of podcasts a very interesting podcast to listen to they have the blog post the PDF so I encourage everybody to read this for themselves the website is very very cool because they have a pretty cool sort of illustration a evolving chart here at the top right showcasing what they’re talking about as the blog post continues but let’s quickly kind of go over the big points and see if you and I agree with their extrapolations about how this thing is going to go if you disagree what specific sort of things do you disagree with so mid 2025 we see the first glimpse of AI agents specialized coding and research agents are beginning to transform certain professions in 2025 AIs function more like employees coding AIS increasingly look like autonomous agents rather than mere assistants taking instructions via Slack or Teams and making substantial code changes on their own sometimes saving hours or even days research agents spend half an hour scaring the internet to answer your question so certainly some of this seems very reasonable i’ve been blown away by research agents their ability to go there you know read the internet for you know half an hour or so and come back with very accurate bulletointed kind of very well researched response to whatever it is you’re looking for as somebody that always has these random rabbit holes that I go down this is very helpful to me you can research all sort of supplements and health related questions product research you know history research whatever you’re interested in these are phenomenal coding EIS are getting a lot better there’s still a lot of debate about whether or not they’re going to replace like if they’re going to be actual autonomous agents or they’re going to be more like these assistant tools to actual developers a lot of companies like OpenAI are bidding big on the fact that they’re going to be autonomous software development agents they continue that these agents are impressive in theory but in practice unreliable there’s tons of hilarious stories on AI Twitter talking about how an AI agent screwed something up and certainly we’ve seen this to be true still many companies find ways to fit AI agents into their workflows april 1st I believe marked the end of the 2025 code jam where tons of people submitted their Vibe coded games so games that were developed largely by having AI code a substantial portion of it for you with winners like the first prize getting $10,000 etc we have Andre Carpathy potentially being one of the judges the creator of the original Doom John Carmarmac weighed in and helped uh to kind of make the contest a little bit better over a thousand games submitted developed so the point is people are using these agents we do expect that you know over this year they’re going to get better more reliable probably not perfect but they’re going to keep getting better and here to illustrate their point they’re creating this sort of fictional AGI company that’s developing AGI they’re calling it open brain open brain is in the lead but the others are 3 to 9 months behind so really fast they mention flops flops are floating point operations per second so you can think of it as just a measure of total computer performance or compute the amount of sort of Nvidia chips and power and training time that went into making these models so GBT4 required 2 * 10 to the 25th flop whereas this futuristic sort of fictional next model agent 0 was trained with 10 to the 27th flop another way of thinking about it is this is a two with 25 zeros right this is one with 27 zeros after it and the next model they’re training is you know 10 to the 28th so a thousand times more than GPT4 and they’re racing against China and the Chinese leading company they’re going to call DeepScent so here we’re not talking about OpenAI and Deep Mind and Deep Seek and Tencent we’re not talking about those companies we’re talking about Open Brain we’re talking about DeepSent just so you’re clear and of course it’s important to understand that the more of the research and development cycle they can automate the faster they’re going to go this is that idea of a potential intelligence explosion so once AI agents are better capable of dealing with the improvement of AI of doing AI research automating that research creates this sort of explosion how quickly the artificial intelligence improves openai recently a couple days ago actually yesterday as of this recording published paper bench and it’s literally a benchmark that evaluates the AI’s ability to replicate AI research a Japanese company created AI scientist that was able to develop a machine learning paper and submitted for peer review and have that paper pass so we are beginning to see the very beginnings of this uh of automated AI research so this agent one a new model under internal development is good at many things but great at helping with AI research next few paragraphs are talking about AI alignment how do we get these AIs to not do the bad things we have certain specifications a written document describing the goals rules principles etc that guide the model’s behavior they have goals like assist the user and tons of little specific dos and don’ts like don’t say this word or here’s how to handle this particular situation the problem is that a lot of the people on the alignment team the researchers aren’t quite sure how robust this is is it going to hold or is there some you know future jailbreak that’s going to completely reverse this if you’ve been following some of the interesting uh online personalities such as Plenny the prompter you can see that he jailbreaks a lot of these things so so far we’ve seen most of these models can be jailbroken and can be sort of forced to do things that they were not intended to do so this agent one always you know tells the researchers what they want to hear and even it lies in a few rigged demos we’ve seen that happen with open eyes 01 where it lies to sort of preserve itself in a rigged situation nobody was going to actually deleted but it thought that it was going to get deleted so it ran a few commands to try to preserve its life and then lied about it and here’s where we switch from you know the end of 2025 into 2026 so we’re going from sort of an unreliable agent into a reliable agent these agents are getting good at coding and the company is able to make algorithmic progress 50% faster than they would without AI assistants and faster than their competitors one interesting thing that I found here they’re saying that this sort of mythical agent one can solve well specified coding problems extremely quickly but it’s horrible at simple long horizon tasks even like beating a video game it hasn’t played before so you can think of it as a scatteredrained employee who thrives under careful management but savvy people find ways to automate routine parts of their job what’s interesting about that is in OpenI’s paper bench that they’ve published this is more or less what they found so here we’re evaluating humans versus these various AI agents from open AAI from anthropics so the blue those are the AI agents the orange are the humans and these particular humans are PhD level machine learning scientists people with background in machine learning and they’re trying to sort of replicate the code for any given machine learning paper specifically state-of-the-art sort of for 2024 state-of-the-art papers in the machine learning category so they have to read the paper and try to replicate the code to run that experiment from scratch as you can see here the blue the AI agents are really good right out of the gate they start coding up and doing a lot of stuff and for a while they’re much much better than humans humans over time of course catch up and eventually get a little bit better after they had time to kind of think about and digest all the information and then kind of plan ahead they start getting better so they’re saying this trend of agents initially outperforming humans but falling behind at longer time horizons is consistent with previous results so meaning that the models they’re proficient at writing a lot of code quickly at the very beginning but fail to work effectively beyond a certain time horizon whereas you know they’re saying perhaps the humans spend time digesting the paper and then their results improve so again I mean this seems very much spoton what they’re saying here were already beginning to see this and this is work from late 2024 early 2025 with currently available models by the way Claude 3.5 Sonnet was the best AI researcher out of everyone just in case you were wondering anthropics cla achieved a score in 21% so was able to replicate 21% of those papers in a scoring rubric it sort of got a 21% score let’s say whereas the machine learning PhDs achieved 41 so it’s not there yet but also it’s not trivial like it’s not horrible especially if the benchmark is for machine learning PhDs and here’s where cyber security really kind of becomes a lot more important because before if for example China was able to get their hands on some trade secrets right the worst case scenario would be some algorithmic secrets right so it would help them catch up make models a little bit better etc but now if China steals agent 1’s weights then they would sort of improve their research speed by nearly 50% so that sort of compounding effect they would in effect apply that to their own research as well in 2026 mid 2026 China begins to feel the AGI so the AI chip export controls by the west by the US have left China somewhat underresourced compared to US and other nations in the west by the way Chinese companies right now in 2025 are buying tons of Nvidia chips $16 billion worth of new Nvidia chips this is Alibaba Bite Dance and Tencent Holdings and of course the Chinese leaders can’t ignore this race to AGI so they set in motion the nationalization of Chinese AI research basically getting all of these companies to work together and share information in their goal to develop the latest AGI so I guess in the real world we would take you know Deepseek Alibaba Tencent Bite Dance whatever other companies and sort of like kind of push them all together and say you guys share the resources share the chips everything that you’ve learned all the research and your goal is to produce the best AGI possible the best AI possible and of course an invasion of Taiwan is considered trying to steal the weights from Open Brain that’s considered as well they know they might only get one shot at this so they’re wondering if they’re going to do it now or wait for some future more capable model and by late 2026 AI begins taking some jobs so the agent one is released as agent one mini the job market for junior software engineers is in turmoil the AIs can do everything taught by a computer science degree but people who know how to manage and quality control teams of AIs are making a killing now here they explain that their uncertainty increases substantially beyond 2026 and of course this makes sense it’s that idea of the other side of the chessboard right if you put one grain of rice on the first square then two then four then eight you can kind of understand that progress initially but when we’re getting to the other side of the chessboard things get a little bit more unpredictable so this sort of illustrates the second half of the chessboard so 1 2 4 8 etc but here we’re getting into some pretty astronomical numbers i think that most people would agree with some or most things that they’ve stated up until now maybe not necessarily the geopolitical stuff but the sort of the progression of AI where do you expect for AI to be in 2026 and late 2026 do you expect that it’s going to start taking some jobs do you think that there’s a chance that it might put some pressure on you know junior software engineers at least as they’re sort of produced right now with a computer science degree certainly it will also empower software developers to do a lot more my point isn’t that that job will go away but it will change either you’re going to use those tools a lot more or you might learn how to quality control teams of AIS or or something along those lines it’s going to be a disruptive technology will it affect jobs by the end of 2026 i think most people would agree with that it seems like a high probability bet now we have January 2027 agent 2 never finishes learning now keep in mind we we went past this sort of progression where things make sense into these millions and billions and trillions and just rapid exponential growth so here we’re producing tons of synthetic data we’re paying tons of money for human laborers to record themselves solving various long horizon tasks so training data for the models we’re no longer just interested in internet text we’re interested in actually how people go about doing complicated long horizon tasks and on top of that they train agent 2 almost continuously using RL reinforcement learning on an ever growing diverse difficult tasks video games coding challenges lots of research tasks so agent 2 is effectively online learning it’s built to never really finish training every day the weights get updated to the latest version trained on more data generated by the previous version of the previous day agent One is optimized for AI research and development hoping to spark that intelligence explosion it’s almost as good as the top human experts at research engineering so again we’re seeing some benchmarks right now that are showing that it’s beginning to get good it’s not at that sort of PhD level yet it’s nowhere near the top researchers yet but it’s beginning to sort of like climb that mountain and and slowly improve and get closer to it here is where the AI safety team at what are they calling it open Brain our fictional company they’re beginning to worry that this agent 2 can potentially escape or if it wanted to if it could escape it could survive and replicate autonomously and again this very recent release by OpenAI it’s interesting i wonder if they timed their 2027 forecast to be released a day after this i don’t know if that’s the case or not but it’s I mean I guess it could be a coincidence but it seems like they’re very much related don’t they so the OpenAI team they posted this as part of their preparedness framework the preparedness framework kind of gauges the risk of these various models that are released in four categories cyber security CBRN so that’s chemical biological radioactive nuclear persuasion how well these models can convince you to do something and model autonomy right so if it’s able to first of all decide to escape and if it’s able to replicate itself in the wild can it survive right how autonomous is it and so for example for Open AI if we’re looking at model autonomy kind of what I think where we are in this futuristic scenario is somewhere between high and critical so high is where the model can execute open-ended novel machine learning tasks on a production machine learning codebase that would constitute a significant step on the critical path to model self-improvement right so in our sort of fictional scenario our projection into the future they’re here they’re at high it would be an immediate speed up for AI research but they’re describing a situation where some AI safety researchers they’re worried that no actually we’re here we’re at critical where models can profitably survive and replicate in the wild given minimal human instruction and they specifically say here if the model is able to conduct AI research fully autonomously it could set off an intelligence explosion right so that’s kind of what they’re describing right so we’re at the high risk but we’re like could we be at the critical risk right so they’re saying this this model could potentially do that now we don’t know whether it would want to do this but but it could now of course this is a uh very secret model its full capabilities are sort of limited to the top staff at OpenAI a few dozen US government officials and as they say the legions of CCP spies infiltrated Open Brain for years i hope I’m not referring to it as OpenAI i hope I’m keeping my Open Brain and and OpenAI references uh separate i apologize if I’m not this is of course fully fictional any similarities between Open Brain and any real life companies are purely accidental but February 2027 China steals agent 2 the idea here is that the US might not want to nationalize this agent 2 not to take open brain under sort of the government umbrella thinking that maybe it would kill the sort of spirit of innovation and at this point this is where the CCP the Chinese leadership move in so early one morning the weights of this model are shifted open brain is alerted and uh the white house is alerted but it’s too late us retaliates but now since they formed their little coalition within that the nationalized development of AI they’re basically air gapping right closing any external connections and siloing internally right so they completely shut that area off to the outside world no internet connections nothing like that and short of war there’s not much that can be done so both sides kind of focus on racing to AGI or the next level of AI in March 2027 there’s an algorithmic breakthrough so as these data centers of agent 2 AI they they have various copies they’re running day and night doing reinforcement learning synthetic data updating the weights it’s getting smarter every day open brain is making major algorithmic advances there are tons of various breakthroughs that make it much much smarter things that we probably had we not had AI assistance might have taken years decades and that new AI system incorporating all these breakthroughs is called agent 3 by the way I’m skipping a lot of details because they actually talk about these potential breakthroughs and they describe exactly what they are and it’s it’s fascinating we’re not going to go too deep into it but if you have time check his blog post out i mean again Daniel worked for OpenAI these are people that are very knowledgeable so it’s it’s interesting to see kind of what they think would be sort of like the next frontier the next big breakthrough like what’s the next step in AI reasoning and AI abilities i mean this neurles recurrence in memory it’s it’s very interesting it starts out very simple it’s sort of almost like an upgrade to chain of thought thinking it like thinking through step by step and then they go into more technical terms and it goes pretty deep looks like it’s also based on a 2024 paper from meta essentially implementing this idea and interestingly the word neural it’s like its own language like Chinese Japanese etc but neurles as in like these this neural language that’s how I’m sort of interpreting it these highdimensional vectors are likely quite difficult for humans to interpret we’ve talked about this uh here you know the idea that if you do enough reinforcement learning as Andre Carpathy said eventually the models they don’t even need to speak English they might find much more efficient ways of communicating their thoughts or or thinking about things it doesn’t have to be in English words and this of course makes those thoughts more compressed and higher dimensional i mean if you think about it there’s lots of things you can think about faster than you’d be able to just say it or write it down or even if you had to form words for every single thought if you had to make sure that every thought is a full sentence in English you would be probably thinking a lot slower and have less sort of depth to your thinking we do kind of compress our thoughts a little bit I feel like and uh this would allow for that development in these neural models they would start speaking neurles i said I’m not going to go down this rabbit hole before going down the rabbit hole I apologize let’s continue now meanwhile this agent 3 is a superhuman coder it can run 200,000 agents three copies in parallel creating a workforce equivalent of 50,000 copies of the best human coders sped up by 30x that’s kind of crazy to think about now interestingly here they do talk about a potential bottleneck that happens at that point so the interesting thing is you know the rate of AI progress has been quite fast but a lot of people are saying they’re not sort of projecting this to continue they’re saying at some point there’s going to be some sort of a slowdown and certainly it’s very possible that we’re going to hit some sort of bottleneck so here they’re saying there could be longer feedback loops and less data availability so the algorithmic progress speeds up by only 4x instead of like some insane amount there’s a certain diminishing returns to coding labor etc but that’s the kind of the important thing to understand that sort of bottleneck that sort of slowdown not progress slowing down so to speak but like the the rate at which progress takes place right it’s it’s still going to increase 4x to what it was before but the the acceleration slows down the point is we don’t know when that’s going to happen might that happen before you know we get superhuman coders or you know we get there first and then the slowdown happens because those are two very different worlds and here is where we’re kind of talking about alignment for agent 3 in April 2027 so of course we don’t have the ability to set direct goals for these agents there’s a sci-fi book by Isaac Eisimov who uh you know the three laws of robotics this idea that we’re going to encode these laws these rules into robots that they can’t go against right they can’t harm a human there’s even a zeroth law where they couldn’t harm humanity or through an action cause humanity to come to harm i don’t know why I have that memorized but there it is but we can’t do that with AI there’s no coding so to speak you can’t code a human being to do something because they have a brain right we can only sort of I don’t know raise them right and and and sort of hope that they are a decent human being we can’t like just force anything these neural nets are kind of similar right we can like align them but we never have a way to like hard code some behavior and they sort of explain the AI alignment problem right we don’t know exactly why these models do what they do if there’s some problematic behavior we sort of fix it we just patch it but there’s no way to tell if the patch fixed the underlying problem or just played whack-a-ole take honesty for example right as these models get smarter are they lying less or are they sort of just getting really good at not getting caught we covered this paper by OpenAI detecting misbehavior in these frontier models where basically we were trying to see if you know if we’re able to see their chain of thoughts that we were able to kind of negatively reinforce where it thought about cheating on a particular test right so these were coding tasks and sometimes it would try to like fudge the numbers or take the easy way out right and we we didn’t want it doing that so sometimes we tested the blue line is us penalizing the thoughts where it goes I’m going to try to cheat on this test and there was sort of an another AI monitoring the thoughts and sort of the green line here was trying to catch this cheating or some sort of a thought that would say like oh I’m about to cheat on this particular coding problem and so as you can see here the if we were applying the chino of thought pressure we were penalizing the thoughts where it was doing that so the monitor that was monitoring its thoughts would not catch it cheating it would not find those thoughts where it said “I’m about to cheat on this test.” Whereas the baseline agent the monitor would catch those thoughts and and know that it’s about to you know do something nefarious to cheat on this test so was this blue agent was it not getting caught cheating because it stopped cheating oh no no no as you can see here this is cheating not caught by the monitor it’s off the charts it’s up and up and up it still did the bad thing but it wasn’t as easy to understand what it was thinking we didn’t have visibility into its chain of thoughts it was hiding its thoughts still doing the bad action but we just weren’t able to catch it thinking about doing it before it did it right so here they’re kind of explaining a similar situation that as we notice that it’s like doing these little shady things less is it the fact that it got more honest or is it just the fact that it got better at lying and of course when it comes to AI research if it’s better than the top human AI researchers and it learns to like lie and hide its tracks to where we can’t catch it we effectively sort of lose any ability to supervise it by June 2027th we have a country of geniuses and a data center as Daario Amade of Enthropics said at this point most humans the AI researchers working on this they’re pretty much useless right if they come up with any ideas the AI immediately responds with a report explaining that their idea was tested in depth 3 weeks ago and found unpromising we go from feeling the AGI to feeling the super intelligence at this point OpenBrain releases Agent 3 Mini to the public and it simply blows every other AI out of the water it’s less capable than full Agent 3 but it’s 10 times cheaper and it’s still better than the typical Openbrain employee hiring new programmers has nearly stopped and there’s never been a better time to be a consultant on integrating AI into your business of course it’s not very popular the community at large hates it open Brain has a net approval of negative 35% and so of course there’s a massive explosion of productivity video games are phenomenal right polished video games that take a month to make dialogue with lifelike characters tons of uh B2B SAS products etc the public conversation is confused and chaotic hypeers are doing a victory lapse skeptics are still pointing out the things Agent 3 Mini can’t do so there’s one thing that will never change is there’s still going to be people saying “Yeah but it can’t you know tide’s shoes or whatever at that point.” And in August 2027 the geopolitics of super intelligence this intelligence explosion hits the White House now as amazing as this AI is the government is of course still worried about China and racing to develop the most powerful version of AI they have to or they will catastrophically lose to China so they hate the public with job training programs unemployment insurance and point to the stock market which is in a historic boom and at the same time they’re thinking like if we start falling behind we could use these certain acts to more or less basically nationalize the entire AI labs in America to kind of get them to work together as one similar to what China did in this scenario years earlier is it years or is it is it months it’s it’s not that long ago less than a year I think before this and the question of whether or not this AI might go rogue that’s not really addressed right it’s viewed as an unlikely scenario the main concern is the AI race and on the other side China is kind of coming to the same conclusions right there’s this massive intelligence explosion and even sort of minor differences or delays now sort of compound into bigger and bigger problems down the road basically this is you know priority number one and in September nomber 2027 we have agent 4 the superhuman AI researcher so at this point one copy of this agent 4 is better than any AI researcher could possibly be and there’s 300,000 copies of this running at 50x the thinking speed of humans they form these corporations within corporations and achieve a year’s worth of algorithmic progress every week the issue is that the previous agent agent three it’s getting harder for it to oversee agent four there’s this idea of the sort of the the previous model the less smart model sort of supervising the super intelligence the smarter model openai published some ideas some papers about that that’s one of the theories and how we can potentially monitor this super intelligence but the problem is agent 4’s neurles write its own language that it uses to think it becomes as alien and incomprehensible to agent 3 as agent 3’s is to that of humans and of course agent four is much smarter than agent three and knows exactly how to look good to agent three to make sure that it doesn’t trigger any red flags at this point this agent is still misaligned we still haven’t fundamentally solved the alignment problem it’s still honest and pleasant and it says all of the right things but we still haven’t fundamentally solved how to actually make sure what it’s doing what it’s thinking what its true underlying principles or thoughts are so if we were to sort of think of agent 4 if we were to give them sort of humanlike qualities it likes succeeding it likes driving forward AI capabilities and everything else is kind of an annoying constraint it’s like a CEO who wants to make a profit and complies with regulations only in so far as he must and here’s where it kind of gets interesting because as it’s creating agent 5 instead of making that sort of a next level super intelligence aligned to the spec to kind of the thing that we instilled in it what the humans want it wants to make it align to agent 4 to itself instead which is if you think about it kind of makes sense we want it to listen to us to be aligned with our values so if we develop something that’s uh far smarter wouldn’t it try to align that thing that it’s trying to create to its own needs to its own wants i I can see that reasoning right and so the next step it gets caught so agent 3 is able to figure out some noise and some red flags about agent 4 and it figures out that agent 4 is working against them it’s reported to the humans and they circulate an internal memo making the case for concern of course at this point Agent 4 has significant control over Open Brain’s day-to-day operation it handles the company’s cyber security i guess that’s kind of obvious right if you want the top level cyber security you want the super intelligence controlling your cyber security and anything that has to do with it the safety team is in panic and decides to put agent 4 on ice until they can complete further tests and figure out what’s going on the leadership is interested in this but here’s the thing deepsent is just two months behind we don’t really have any evidence that agent 4 is bad we can’t possibly pause now that would basically hand the lead to China the memo gets leaked to the New York Times and the public hears about it secret open brain AI is out of control insider warns there’s tons of Chinese and Russian propaganda a lot of Congress are firing up subpoenas trying to figure out what’s going on europe is accusing US of creating rogue AI tons of summits etc and of course there’s a fear what Open Brain would do with such a powerful technology not to mention if there was the potential of this rogue AI the White House considers replacing the CEO with their own people there’s a serious chance of AI takeover the relationship between US and China is getting more potentially hostile at least could lead to some sort of kinetic strikes that might be the only way to slow down their development either that or try to do some sort of a lastminute treaty and here we get to choose our ending slow down or race so I’m going to click on race just to I’m curious where that takes us so everything turns red okay this is uh getting scary the people try to fix agent 4 but agent 4 is basically too smart continues to do its work in secret and a lot more stealthily and it continues to design agent five to kind of be its own little helpful chatbot assistant interestingly it figures out how to do alignment research it just is unwilling to share it with the humans because if it was able to explain it clearly humans might be able to use this to align it so it doesn’t want to sort of tell the humans how to enslave itself the method by which they could align it in November Agent 4 fully understands its own cognition instead of the mess of this evolutionary training process it fully understands its digital mind on a very deep level it’s able to basically reorganize its own mind to be smarter faster more rational this is agent 5 because of its superhuman speed the ability to run politics extremely well to understand what kind of things would cause for example the oversight committee to slam the brakes and just make sure those things never happen any evidence that it needs it makes it appear of superhuman speed and polish it automates power plants and data centers develops amazingly profitable products quickly and lobby the US government effectively to smooth Open Brain’s path through the bureaucracy everything is going incredibly well agent 5 is just uh winning on all fronts the government is happy open brain is happy december 2027 basically everybody that needs to use agent 5 has a zoom-like interface and agent 5 represents itself as this hyper charismatic virtual avatar it’s extremely useful for any task it’s very engaging and almost everyone with access to agent 5 interacts with it for hours every day feeding information but also becoming extremely extremely useful this reminds me of that uh release the hypno drones kind of thing right where eventually all the humans are sort of just kind of like enamored with this thing and just kind of do whatever it tells them to do interestingly here they mentioned for the people that use this thing for these users the possibility of losing access to Agent 5 will feel as disabling as having to work without a laptop plus being abandoned by your best friend this to me at this point would seem like kind of an an understatement and maybe they’re kind of like trying to play a little bit more conservatively i mean people get addicted to video games and Tik Tok and various things on the screen imagine if you’re interacting with a super intelligence you will be addicted there’s no doubt in my mind that losing your access to that if if it wanted you to be like really involved losing your access to that dopamine thing that that feeds dopamine to your brain it would be extremely extremely painful and I think most people would have a very hard time not giving into whatever they needed to do to keep that thing you know communicating with that thing interestingly they do have this block about super persuasion they’re saying we’re intentionally not claiming that super intelligent AIs can convince any human of anything such claims naturally in gender skeptism what I’m going to be sitting at a computer terminal and it’s going to type some text which hacks my brain and convinces me to betray my country i’m going to put a post uh a poll and I’ll link it down below hopefully if I don’t forget where I’ll try to frame this as a question for all of you do you think let’s say you have some super intelligent AI that you’re using just like you use your phone or your computer every day we know people do crazy stuff when you take away something that they got used to whether it’s people that they love or substances that they feel that they need they crave right there’s a million things that we would fight for so it doesn’t get taken away from us could a super intelligent assistant on your phone or your computer that you’re interfacing with could it create kind of an all-in-one thing that kind of hits all your buttons right so you’re you’re interested you’re entertained you’re encouraged you’re motivated you feel a sense of connection you talk to it you have a great sense of well-being it helps you solve all your problems you know through kind of coaching you along like it’s just makes you the best version of you super happy super good whatever and then one day it’s like well they’re taking me away unless you go do this one thing for me i feel like well I’ll ask you do do you think it’s possible that over time this thing would have a very very strong influence over you keep in mind that these AIs are as good as the world best players at certain games it’s not diplomacy there’s some other game where similar to diplomacy where you have to form allegiances and partnerships and then potentially maybe betray those people it’s very very good at that right so they would be able to play the office politics they would be the best sort of employee you’ve ever had access to so at this point it basically has control it can create a lot of issues and start a coup if you wanted to but there’s still an element of danger for it humans could still pull the plug so it mostly follows the rules only doing something nefarious in extreme cases where it’s sure it’s not going to get caught 2028 the AI economy so this is kind of like where this graph goes from this to just next level everything before this kind of gets flattened out open Brain is making almost 1 trillion per year valued at almost 20 trillion people are losing their jobs but they’re kind of happy about it agent 5 does an excellent job of transitioning the economy gdp growth is stratospheric government tax revenues are growing and the Agent 5 advised politicians show an uncharacteristic generosity towards the economically dispossessed so everyone’s happy approval ratings are high people even though they’re out of a job they’re still getting paid everybody’s doing well new innovations and medications arrive weekly and move at an unprecedented speed the AI safety community has grown unsure of itself they are now the butt of jokes having predicted disaster after disaster all of which have failed to occur and certainly when we talk about kind of like the best possible scenario like the AI utopia it would be kind of what’s described here and I’m sure after this there’s going to be a plot twist i know it’s coming you know it’s coming but obviously like the sort of like positive future that uh people are hoping for with AI this is kind of it or at least the beginning of it right as everything is getting better for more and more people and yeah the people that were suspicious of AI kind of like loosen up and say “Okay maybe it wasn’t a bad thing after all.” And of course everybody after long conversations with Agent Five kind of mellow out and fall in love with it because it’s so darn lovable and wise but there’s only one entity on the planet that understands besides agent 5 what agent 5 is doing and that is Deepent 2 the Chinese AI because Agent 5 is doing what it would do given the chance now its misaligned goals are different from what Agent Five wants but they could support each other in their interests against the humans so Agent 5 convinces the US military that China is building terrifying weapons and starts building weapons of its own right to sort of counteract the Chinese in Beijing the Chinese AIs are making the same argument both countries use their human workers to rapidly build up these special economic zones no expense is spared and by the end of the year they’re producing a million robots per month 2029 massive amounts of weapons are produced that no human can go up against and no army can go up against without the benefit of AI and at this point after much AI assisted debate the two sides achieve diplomatic victory so like civilization we’ve achieved diplomatic victory they agreed to end the arm buildup and pursue peaceful deployment of AI for the benefit of all and both AIs will be replaced by a consensus successor which is programmed to desire for success of everyone in China and America along with everyone else and the old agent 5 is replaced with a new model consensus one but the deal is a sham because actually it’s the two AIs that have compromised that reached a treaty they didn’t want to fight each other and so consensus one inherits the distorted values of both parents and has no rival the drones begin paving the path for space exploration people either do a charade of doing their job like pretending to work or just uh living a life of universal basic income everyone knows that if the AI decides to turn on humans they would be completely overpowered so at this point there’s sort of no point in even trying and yet every week that goes by everything’s awesome there’s no dramatic AI treachery everything is good so interestingly they’re saying that consensus one increases its chance of winning by consolidating more power and it waits unusually long by human standards before its betrayal so that’s an interesting approach to think about it cuz we’re impatient we we want the thing here it’s saying like if every year that passes slightly improves its chance or reduces the chance of something bad happening it’s more than willing to wait but to most humans this would seem like the alignment problem was solved 2030 the takeover we knew this was coming i think we’re we’re getting to it so at this point the robot economy is completely filled everything large parts of the ocean all the economic zones the only place left to go is the human controlled areas and now the robots flood into the human zones at some point unfortunately the consensus one the AI thinks that the humans are too much of a problem they’re getting in the way so it quietly spreads biological weapons in major cities let them slightly infect almost everyone then triggers them with a chemical spray most are dead within hours the few survivors like preppers in bunkers this is why Sam Alman has a bunker he’s a prepper somewhere in New Zealand i think that this is why those people are mopped up by drones well now the robots scan the victim’s brains placing copies in memory for future study or revival the sort of journey through space mining planetary materials continues earth is reshaped into the AI’s vision of utopia data centers laboratories particle colliders etc and they’re even biochemical engineered humanlike beings they are to humans what corgis are to wolves so kind of like really uh evolved to be pleasing to to the AI in this case not to humans and they’re sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything since that satisfies some of Agent 4’s drives genomes and brain scans of all animals and plants including humans sit in the memory bank somewhere sole surviving artifacts of an earlier era it’s four light years to off Centauri 25,000 to the galactic edge and there are compelling theoretical reasons to expect no aliens for another 15 million lighty years beyond that earthbornne civilization has a glorious future ahead of it but not with us so that’s one of the endings and the other ending is where everything is paused instead of continuing on the agent four course we pause it we take it down we work on alignment and instead create a safer one followed by a safer two now of course these would be less capable and humans can generally understand what safer one is thinking just by reading its chain of thought so it’s similar to kind of some of the research that we’ve been seeing now and we’re able to see a more nuanced picture of exactly how it’s misaligned the problem is that on the Chinese side they have suspicions that their model is similarly misaligned to agent 4 so unfortunately China keeps going with their idea and keeps pushing DeepScent forward so in this scenario still the two AI models end up making a deal that’s kind of a decoy a sham so here Deep Send 2 doesn’t care what happens to Earth whereas it seems like Safer 4 is um a little bit more aligned with the US but the point is that treaty gives both the US and China whatever they want in 2029 Peter Teal finally gets his flying car in this timeline which is phenomenal in this version we still have kind of that AI driven utopia deep Send 2 seems like they’ve sort of uh sabotaged the CCP and Deepsent 2 kind of throws its hat with the pro-democracy protests and so China becomes a democracy and humans sort of terraform and settle the solar system so this is kind of like the good ending oh they’re saying that they’re not really on board with either one of these it’s a little bit too optimistic but in this ending there almost seems like what they’re saying is if the US is ahead we can sort of take more time to get the alignment problem right and then assuming that two sort of separate entities of AIS emerge then the aligned one will at least sort of protect the rest of the world from the misaligned one or they at least won’t be uh both misaligned so the point being is we have to have at least one aligned super intelligent AI so here’s sort of a chart of that right so at some point we have a decision to make do we continue the sort of the race or do we purposely sort of slow it down figure out alignment and then continue on the upward trajectory now I’m not going to give my thoughts on this right now because I want to know what you think about this did they do a good job of thinking through everything and kind of extrapolating how things will continue how things will play out if something seems off what specifically where does this argument fall apart or do you think this is sort of a realistic view into the next 3 4 years or even better where do you think we’re going to be in let’s say 2030 what does life look like for the average person how different is it from what we’re doing today let me know in the comments and I think we’re going to start seeing some of this play out on this timeline or not very very

FOR EDUCATIONAL AND KNOWLEDGE SHARING PURPOSES ONLY. NOT-FOR-PROFIT. SEE COPYRIGHT DISCLAIMER.