Learn more: The Institute of Art and Ideas (IAI)
hi my name is ganes Taylor and I’m your host for AI and the end of humanity from sophisticated robots to the development of new drugs artificial intelligence or AI is shaping our future in many sectors once we thought the creation of original text and forms of art were the select Preserve of human beings but the exponential development of AI is challenging these assumptions an increasing number claim that AI threatens the very idea of what it is to be human a recent survey of more than 900 technology pioneers and policy leaders predicted that AI would threaten human autonomy and agency with over a third saying we would be worse off in the future confirmed they argue by musk’s proposal that we should link our brains directly to machines so should we see AI as opening up an era of groundbreaking Innovation or does it foreshadow the loss of vital human attributes and Independence or does AI fundamentally Challenge and change what it means to be human or is all of this talk of its radical importance a sign that we’ve actually just been taken in by the marketing hype of an enormously profitable industry and humans not only remain very much in control but will continue to do so going forward we’re joined by four fantastic speakers alaza yudkowsky is a leading AI safety researcher who focuses on super intelligence and runaway AI he has called for a stop in AI development to avoid catastrophe Scott arenson is one of the top Quantum uh complexity theorists in the world renowned for his research in Quantum Computing Scott is now working for a open AI on the theoretical foundations of AI safety livb is one of the most successful poker players of all time since retiring from poker Liv is now a leading figure in the effective altruist MO movement she too has warned of the severe consequences of runaway Ai and last but by no means least we have yosa who is a cognitive scientist who is pushing the limits of what we can achieve with AI renowned for his philosophical Insight yosi’s Research delves into the similarities and differences between the human mind and artificially intelligent systems so thank you so much to all of you for joining us Ela if you’d like to take it away two minutes on the question of whether or not AI fundamentally challenges what it means to be human I have no idea what it could mean for something to fundamentally challenge what it means to be human the meaning that we bring to humanity is something that we bring ourselves what some things outside of ourselves should not be able to change it very much could AI change who we are well if we uh knew how to shape AI if we knew how to get nice things out of AI if we could get magic cornucopias out of AI and if we knew enough we probably could do that when we could change who we are that would change what it means to be us this seems hard I think the actual Challenge and it’s not a very fundamental one that AI poses to us today is how do we back off we are not ready uh this looks to be a thorny thing that is difficult to get exactly right and it’s at a power level where if we don’t get it right it just smashes everything um our institutions are nowhere near capable of handling this um it’s not that there’s a near Miss in progress this is you know they’re just not even coming close to what they need to do to get things under control so I think we are faced with a unprecedented World level challenge of don’t have anyone anywhere do this that was very clear Scott what’s your position on this I mean if yeah yeah so so I also don’t know what means to change what it means to be human but uh uh I I think that absolutely uh uh AI could challenge our sense of uniqueness as humans as you know you know the only uh uh beings in town uh who uh who can uh uh understand mean things I think to some extent that has already happened right we are uh as of a few years ago uh no longer the only entities on Earth that can uh uh that that can have a detailed conversation about you know a given topic uh uh with at least the semblance of understanding that topic uh we now have llms uh that can do that and you know we can still see differences we can still see uh things that we can do that uh gp4 for example cannot uh you know it can’t yet uh uh write my research papers for me or you know nor can it uh uh uh change a light bulb or you know all kinds of manipulations of the physical world but I think that we are you know now uh uh confronted with a question that uh uh you know generations of Science Fiction writers grappled with you know before us of well well what happens uh when we get AIS that can do just about anything that we do uh uh as well or or or better than we do it uh you know sort of what what does it mean to be human in that kind of world you know I think that starts to change not merely kind of what advice do we give kids about you know what careers to have but uh what is even the point of life at all you know it then uh uh feels you know less like uh uh you know producing something that only you could have produced and more like uh um um you know Finding specialness and just the fact that you know maybe well you know even though an AI could have done something better I wouldn’t have done it exactly the same way as you you know this is your uh poem or your music sick or or something like that or or just sitting back and enjoying life which uh probably is already the goal for a very large fraction of people and maybe the small fraction for whom it isn’t should just get over themselves fantastic Liv I mean where are you at on this question of AI challenging what it means to be human uh so yes I do think AI will challenge what it means to be human because what it means to be human has never been fixed it’s always been changing our technology and environment changes over time and that’s fine I mean there’s no guarantee that though that this change will be good for us uh like I think we can all agree social media has created a bunch of undesirable outcomes like making us more narcissistic uh more tribal and just decimating people’s attention spans and the thing is it’s not like Zuckerberg and Co wanted to do all of these bad things it’s just that it so happened that market forces best rewarded the platform forms that designed their algorithms this way and frankly I think it’s what anyone who thinks it’s sufficient to entirely rely on markets for product safety is a naive idealist markets don’t directly select for human well-being they select for products that appear sufficiently good for human well-being such that they can be sold and so if you can find a way to quietly externalize your cost then that’s even better and so the result result of this subtle difference is that we end up with a bunch of tragedies of the commons and in some cases even direct deceptions like when tobacco companies pretended for decades that cigarettes don’t cause cancer um or when Volkswagen cheated on emissions tests uh or more recently uh when 3M uh got their 10 billion fine last year for leaking pfas into the water supply uh they leaked these when they knew that these forever chemicals are likely to be carcinogenic but again Market forces motivated them to keep on doing to using these chemicals anyway as long as they could get away with it so sadly history has been littered with Psychopathic corporate Behavior like this and it doesn’t seem optimal to me at least to have the same mechanism that allows for or even encourages these behaviors to be the sole driver of advanced AI but that is our current trajectory right now and Ai and especially AGI is is set to be a bigger Paradigm Shift than the agricultural or industrial Evolutions revolutions so we need to treat this change to humanity with the care and wisdom it deserves and prioritize getting this change right rather than just doing it as fast as possible fascinating okay yosa your your two minutes please I think that to be human means to challenge how what it is to be human right it’s a difficult question what what it even is we are some kind of wind that merges in the activation patterns of neurons that talk to each other and in a way AI gives us New Perspectives on how to deal with this condition and um I agree with the other speakers that AI is a technology that changes the way in which we are interacting with ourselves and with each other and how we look at ourselves of course um I think that without technology um we would not be able to feed the populations that we currently have the markets that sometimes produce bad incentives are also the only thing that is has enough productivity to feed the people that are on the planet if he abolish these systems and replace it by anything else that has been tried so far we would dramatically have to reduce the number of people on the planet or that it would reduce itself because we without for instance the chemical industry we wouldn’t have nitrogen binding and without nitrogen binding we cannot produce a lot of food and uh when we are currently on a trajectory where we basically need better technological um solutions to understand how to interact with our ourselves and with the planet to stay on this planet and without AI I suspect we will be in dire spot AI is not solving all our problems but it’s throwing many of the balls up into the air and increases our chances of survival at the moment I also don’t think that social media are bad in the sense we are a society that is very confused or syation is confused and incoherent and social media allows us to become more coherent and the first step is that it allows us to see how incoherent we are right before that that we only had to rely on media to tell us about reality and now we realize that we cannot trust media we knew this before but we had no other ways of becoming coherent and this is a process that takes time we’re still in the early days of these developments and I think AI is an important building block of making us more coherent okay that there’s a lot of food for thought there I think it might be quite helpful for us to sort of um start relatively basic let’s say so I heard um llms IG uh IGM various uh letters in combination so let’s just let’s just start by by acknowledging the fact that this thing that we call artificial intelligence has a long history of being developed and that uh we are now in this sort of Renaissance Era where the systems that that have been being built by human beings are becoming recognizably more intelligent is that a fair starting point let’s say you can all just nod or disagree or whatever yeah is that about Fair perfect okay so I guess the the the Salient Point here is you know it seems that what we now call AI and recognize as AI has gone over some sort of threshold where we start to recognize sort of human characteristics or Proto human characteristics within what these um systems are doing and so I guess the question is will AI systems continue to develop in that sort of trajectory and become more um uh recognizably carrying of human characteristics maybe even of understanding and then ultimately emotions and I think it’s the Consciousness part of it that really gets people sort of gives them the heebie gbes so Scott I wondered if you could sort of um talk to that a little bit this idea of you know AI is developing it’s it’s getting better and bigger and more recognizably human and is it actually possible for it to develop understanding or emotions or is it possible to do any of those things even well first of all I would say that that the large language models that we have today you know they they already give verbal responses that if a human had given them to you right you would have said yes there is emotion there there is understanding there I think that you know you would have ascribed all of those things to it uh so you know in in other words uh I would claim that uh the we already have uh as of a few years ago AI that pass some version of the touring test you know it it depends you know always on on who is the judge and and what kind of topic are they allowed to ask about and so forth but uh you know I think that uh in in normal conversation you know these things you know they’re they’re they’re um you know um not only do they pick up on emotional nuances but you know because of the way they were trained they often do that better than humans still right and uh and and ironically you know this is something that almost no one would have predicted uh 10 or 20 years ago but uh they are in many ways better at emotional Nuance than they are at at logical reasoning you know or at things that that you know you you would have thought were uh the strength of a of an AI so um uh it’s clear that these systems are going to continue to improve you know there is a you know Ian I mean the way the main way that they’ve improved as much as they have has not been any radical new idea that we didn’t know you know in the 80s or the 90s right you know incredibly it has mostly been scale right it has been U you know building models that have have you know many billions of parameters or trillions and then training those models on you know a large fraction of all the data on the internet you know that is what has given them either you know intelligence the semblance of intelligence whatever you want to call it and uh and there is a clear path forward to just keep building bigger models and training them on more data now we don’t know uh uh how far this will continue for one thing you know we’re as we scale the models we’re going to run at some point into uh uh energy constraints you know the the fraction of energy that you’ll need will be a large fraction of all the energy that’s produced on Earth that’s why you know Microsoft is investing in nuclear plants now for example right there’s also the question of you know you know uh uh where will we find more training data right I mean you know we’re already using most of the text on the internet uh there’s all of uh Tik Tock and Instagram that could still be fed into the mall but you know that might just make the models Dumber rather than smarter right so so so it’s not clear whether this will ASM toote or some point or whether it will uh reach you know a human level you know and of course human level is not at all the a natural Terminus here right you could also go beyond human intelligence Beyond human emotional understanding you could or you could you could imagine that but uh it’s not clear whether that’s going to happen with existing uh technology uh uh you know with with the existing approach even if it doesn’t of course it’s entirely possible that someone will come up with a new approach uh that will get around the limitations of this one and then finally you asked about Consciousness so that’s the that’s the hard metaphysical question because for two you know 2,000 years uh no one has known how to reduce that question to anything empirical right you could have a model that uh uh is interacting you know uh uh perfectly like uh uh like like any human that that you would name you know no one can tell the difference and someone might come along and say well okay but because it’s made of silicon and it’s not made of meat you know we know that it’s just it’s it’s it’s it’s just a bunch of ones and zeros it doesn’t really understand anything right and of course there’s an obvious rejoinder that that you know that that that those critics almost never want to Grapple with which you know well how do we know that you really understand anything I mean you’re just a bundle of neurons and synapses right but now you get into a philosophical problem that I think some of the greatest Minds have argued about for thousands of years we’re probably not going to resolve it in this conversation uh but but I but I think uh uh C can you have AIS that would at least produce the external behavior that that in humans you know we would associate with Consciousness to some extent not only is that possible but to some extent we already have that I would say I mean Phil does anyone disagree with that all right so the current set of systems are being trained to predict what humans will say and do individually not just what humans on average do but what any individual string of texts on the internet will say next um they aren’t being trained to imitate an average human they are being trained to predict individual humans suppose that you find the finest actress you can set her to watch a Tavern for months at a time until she can predict what every single person in that Tavern is like individually as they get drunk and convincingly put on individually perfor individual performances for what each of the people in The Tavern are like when they are drunk is she drunk herself probably not because to predict a drunk person it doesn’t actually help to be drunk yourself to get forget for yourself to get disinhibited yourself to like put on a really perfect individual performance smash the individual you want to bait your top performance for that you don’t want to be drunk the current AI are being trained to predict what humans talk about including what happens when humans talk about Consciousness if you wanted to find out whether they were spontaneously expressing a sense of their own existence you would strip all the talk of self-awareness and Consciousness out of the data set they were being trained to predict and then see whether they talk like that inter um as far as I know nobody’s even expressed any interest in running this experiment um because you know it’s not going to make money right imitating OT many people express interest but they haven’t doesn’t mean they’ve done it cool all right they’ve expressed interest but they have no corporate ability to actually run the experiment because it’s not going to make money putting on convincing performances of emotion sounds like something that could make a lot of money you have the the image the video of this human-shaped figure who seems to really be interested in you and maybe even be in love with you that sounds very profitable so there’s also a certain C amount of profit motive in continuing to maintain your current state of having no idea what goes on inside the system just vast football like hundreds of thousands of football fields size spreadsheets of inscrutable floating Point numbers and as long as you the company don’t know what they mean you can tell people well for all we know the AI really loves you but keep in mind that what the AI has actually been trained to do is predict exactly what individuals will say next including when they’re talking about loving their Partners I mean I would like to add that I mean further to that we we as you say we we haven’t there doesn’t seem to be anywhere near the sufficient amount of effort or resources put into coming up with good tests of Consciousness like one of the best ones I’ve seen thus far was just by a guy on Twitter um this guy Josh Witten did a thread where he did like the Mir a test with a bunch of different llms and you come away from it and it’s like I don’t know after reading that I slightly update in the direction I mean only very slightly again my era bars are enormous slightly direct update in the direction of them being conscious but the point is is like in either either we end up under ascribing Consciousness to something that actually has developed it or we end up giving something conscious like falsely ascribing Consciousness to something that absolutely doesn’t deserve it and thus giving it a bunch of resources and rights and so on that will come away from Human humans in at least in the beginning both of those could be enormous moral catastrophes but no one seems to care they’re just like let’s go as fast as possible who cares la la la la la like I would like to to see some fracture like I mean I I’m sure the fraction is greater than zero that is being put into coming up with some Consciousness tests but it’s certainly not more than 1% in terms of resources I’m sure Scott you can talk to this more but like it I to me like deserving of 20 30 something at least percent um so yeah it’s vastly beyond their present abilities they can’t appearance the systems they don’t know how they work how are they supposed to test it right I I I I guess a a test for Consciousness is the kind of thing where how would you even know if you had succeeded right well do saying goes that basically robot needs to be able to recognize themselves in the mirror and hate what they see then that will be fully [Laughter] human but uh I think that we need to address this question of how to make sense of machine Consciousness in Earnest and um what I’m currently proposing is that we build an Institute for this purpose it’s a nonprofit we probably should do it in San Francisco because everywhere else it would be just an art project and um stting an initiative across companies Academia and the Arts um to deal with this question in Earnest and um will you tell H how will you tell whether you’ve succeeded um we find this out in this but in N sense I currently my hypothesis is that Consciousness is um when we pointed it um indexically two things it’s a second order perception you notice that you’re noticing and that might be an artifact of Consciousness being a self organizing process so it needs to have some kind of uh control mechanism that establishes its own nature and keeps it stable and the other one is that Consciousness creates now and happens now creates this bubble of nness that we subjectively inhabit and so uh based on these observations i s suspect that it’s something like a consensus algorithm that basically makes all all our models coherent and this bubble of coherence that it creates and that we observe is what we experience is happening right now and this uh self-perception is part of this bubble and that’s why it’s it’s present um this reflexive nature of Consciousness bu in this bubble and so when we ask the question is a cat conscious this question for me comes down to the question is the cat aware of the fact that it’s aware so basically is the behavior of the cat it’s external and internal behavior of the cat driven Cally by the representation of that awareness of the cat of its self-awareness and uh subjectively I have the impression that it is because I can also vipe with the cat I basic can go into a state where we have perceptual empathy and the cat and me have Bas bu feedback glops through each other and notice that the other one is noticing aspects of the mental state of the cat now that is something that the LM cannot do because it’s not uh perceptually coupled to the environment to me uh perceptual coupling of such system is a very interesting question I also uh think that if we want to do such research it makes a lot of sense to do this uh up to cut level and uh because if you are building systems that are at the human level it’s very difficult to take responsibility for their actions presumably if you go fast the superum level at the civilization level and so on the BS are again up in the air and it’s very difficult to make that discussion again but um I think at a human level already we are in a situation where it’s difficult to take ethical responsibility to go back to what Scott said at the beginning so I found it really interesting that you said that effectively through the creation of the internet it was possible to effectively generate a data set so vastly large that previous artificial intelligence sort of approaches were able to actually Reach This level is something that we can sort of at least partially recognize as being a mimic of some kind of human behaviors right and Ela you were saying that um it is just that it is a mimic it is a personalized mimic to to a prediction sorry you were saying right a prediction of what’s going to happen next I’m saying that’s how it’s trained right What mechanisms perform the prediction what actually goes on the side and what it’s like to be a large language model are something into which we have no vision right our our glasses are blinded right so we do we understand the externals that doesn’t mean we understand the internals right okay so what we’re seeing is something that resembles a sort of predictive mimicry of what we’re expecting but we don’t actually understand where this thing is coming from and to link back to what you were just saying yosher I found myself thinking so the question was will it develop all of these kinds of characteristics right we’re trying to make a prediction here about how we think these things are going to go and I found myself thinking perhaps because of my own background in developmental biology but you know human beings uh don’t start off as particularly conscious I don’t think right I mean I’m no child psychologist but there’s a there’s a sort of Developmental trajectory to human consciousness as we recognize it even and so I’m curious as we sort of wrap this theme up whether or not we we think that it’s almost an unfair question to ask like you wouldn’t ask a 2-year-old is it conscious or maybe a one-year-old or something like that and perhaps these AIS are sort of at that level and and so then we get back to the same question again which is well do we think it’s actually possible for this thing let’s say we overcome the barrier of physical energy let’s say we overcome the barrier of how much silicon we have on planet Earth say we feed it every single piece of information from every single human being on Earth how how far can this thing actually go I suspect that’s a misunderstanding of course a two-year-old doesn’t have a lot of metacognition so it’s difficult for a 2-year-old to conceptualize the fact that it is conscious but I don’t think that we become conscious after the PHD we become conscious before we can track a finger and we observe that those humans which do not become conscious remain vegetables it’s the prerequisite for Learning and organ organizing the brain and so I suspect because everybody becomes conscious in order to achieve uh the performance That You observe in human that doesn’t seem to be any kind of alternative and any kind of animal that we see that is complex uh that Consciousness itself is relatively simple and it’s a learning algorithm it’s the simplest learning algorithm that nature has discovered to train the self-organizing brain so of course there is not a self at this point but the self is not the prerequisite for Consciousness the self is a Content that Consciousness creates to model the person that interacts with the world there are there are people who believe that there is something special about the biological substrate that that we are running on that you know allows us to have firsters experience and it would not allow an AI to have something analogous to that no matter how impressive it Its Behavior was now I don’t know how to prove those people wrong uh uh but I would also say that that the burden is on them to articulate you know what is special about you know the uh uh uh wetwear as it’s called about the you know biological uh Hardware that that we are running on but you know but that that question now you know is is no longer the domain of just you know uh philosophy and science fiction right uh if you ask me like like you know the only thing that that that I can see that really fundamentally distinguishes you know our brains as we currently understand them from any digital computer is that you know digital computer programs are copyable they’re backup we have perfect visibility into you know what every uh bit is doing at every time step and you know with our brains we just uh uh don’t have that you know we don’t know if there’s a fundamental physical limitation to learning the exact state of someone’s brain but you know I I think uh if you wanted to say that there was something uh uh uh about a brain that could that couldn’t be replicated in in silica even in principle that you know you would have to attach great importance to those things I think that I would to point out there is like one consistent theme throughout our history of dealing with Consciousness is that we have a very large tendency to undescribed it to other types of humans long ago then it took us a long time to give or at least whether it’s Consciousness or like moral personhood we wouldn’t give it to other humans we wouldn’t give it to we wouldn’t we thought animals had zero ability to suffer and or like even when we figured out they could suffer we still didn’t think it mattered Etc and over time so we always lag behind and so we should bear that in mind in our that that was precisely touring’s argument you know 75 years ago and yet we’re still relying on his on his like napkin side note like that’s the one thing we’ve got during on a political level suppose that a senior looking scientist comes and says I have a theory of Consciousness I understand Consciousness it’s not the consensus of philosophers of course but I understand it Consciousness requires higher order reflection you need to be able to model yourself and you need to be able to model yourself modeling yourself and when you talk to a human 2-year-old you can tell that they sure don’t have second order reflection going on in fact if you look at like how their brain develops and they start going on brain you know like this thing is developed that thing developed therefore says the scientist I am very confident that human 2-year-olds have no qualia and it’s okay to eat them I think that a voter is incredibly valid if they say maybe you’re right but how can I trust you and therefore it’s going to remain illegal to eat two-year-olds what about large language models why believe the guys at the AI company anymore than you’re believing the scientists well what what what if the law becomes you’re allowed to delete a large language model as long as you keep a backup copy copy somewhere else would you apply that policy to two-year-olds well how do you make a backup copy of the two-year-old if if you could then uh maybe I mean this gets into you know the old chestnuts about would you uh uh agree to use the teleportation machine that works by just destroying the original copy and and so forth but the question whether animal doesn’t hinge on whether they have qualia or not they quite clearly have qualia there are other considerations that come into play it’s a quite complicated question which rights are being given to which system under which circumstances but uh there’s another issue I would like to point at the autological status of Consciousness is not a physical object it’s not a a thing that we can put into a petri dish or under a microscope or measure in this way because it’s a simulated property it exists as if it exists virtually it right our we look into the brain we just see activation patterns and it must be something in these patterns it’s it’s a pattern this CA of power and if you think about this fact that the person that we are is not real point that meditators and philosophers has point to across cultures for as long as we have history about this that the self is a construct that is a story that the mind tells itself and our Consciousness itself can be deconstructed because it is a representation once we take this into account the question of whether the simulation that in LM makes off a Consciousness is more simulated than the one that our brain makes becomes quite hairy it’s still interesting question that has answers that are meaningful and I don’t think that the llm is conscious in the same way as we are but uh it if it Consciousness is a simulation and so on the question is at which point does the simulation in the machine become functionally equivalent in the in the ways in which they matter to us also why does that matter sorry this is the thing I’m not getting here so you know in terms of the answer to the question will they be able to develop all of these things and will they become conscious we’ve gotten stuck on the how do you prove it but let’s say they do right so what like what is the concern here that people have with this idea of AI developing you know abilities to have Consciousness and what is it just a fear of losing control of something that we’ve created well you can lose control of something that doesn’t have your own style of Consciousness but if it does have your style of Consciousness it can hurt and then you ought to not hurt it and if you say well I don’t care cuz it’s no off my back then I’m like I care because it’s a it’s a fellow sensient being and perhaps we just have a political conflict there but mostly I suspect that you’re not really integrating what it would mean for there to be somebody inside a computer who was hurting yeah maybe you know like if if you found out that you were somebody like that would would would it suddenly be okay to you for other people to hurt you I I do think that it’s crucial to separate the two questions of one you know what kind of internal experience if any will AIS have and secondly what kind of effects will AIS have on the world right they could change the world they could even replace us you know uh conceivably even without either without having any Consciousness or without having any Consciousness that we can understand by analogy to our own case well I mean I mean uh if if you’re just asking what kind of effect you know something will have on the world right now this is an empirical question this is a question about predicting what will happen right and you could postulate an entity that is smarter than us that is motivated to you know uh uh uh uh control the world you know that is uh that basically thinks of us the same way that we treat that we think of orangutans or or something like that and and you and and and and and that would seem frightening and and the question as to you know does it have an internal experience of consciousness comparable to ours that that that doesn’t even necessarily arise in predicting what it’s going to do or not do and and I would say the question of what Consciousness is and how it works is the most important philosophic question at all so I don’t really understand K how you cannot be curious about this question of course I’m curious about it don’t want I’m just saying questions okay okay hold on so there’s obviously a lot going on it’s great to to have this kind of thing and it’s not that I’m not curious just to be very clear but my question here is about actually like materially what do we like what what does an AI catastrophe look like because there’s a difference between is this thing feeling stuff are we going to hurt it which was your point elasa right like is this thing conscious will we be causing it harm wouldn’t you want that to be you know wouldn’t you want that to be extended to you that’s one question but it seems that most people’s concern is how is this thing going to hurt me and and that’s like an elephant could squash me to death and not know that I exist existed what difference does it make if it has Consciousness or not or is it because we’re worried that this thing might actually understand what we have done and what how we live and pass judgment on us and then choose to squash us neither of those things it knows you’re there but it doesn’t care it’s not that it hates you it’s that it wants its own thing and you’re in the way it it knows perfectly well that you’re there it knows that you’re conscious it doesn’t care about whether or not you’re conscious and that’s how it ends up steing um because if you most things that an entity can care about are not you humans not even Consciousness they could care about tiny little spirals they could care about tiny little rhombuses they could care about enormous complicated clocks they could care about chocolate chip cookies they can care though that one’s pretty unlikely but s giant mechanical clocks and they could care about many many combinations and lots of lots of these things um but the most ways you could randomly describe a computational predicate for valuing something it’s not going to have sensient beings Happ having fun living happily forever ever after as its Optimum now this isn’t to say that AI is necessarily that all possible computer-based Minds necessarily don’t care about us it’s my prediction for what happens when people build them at their present level of competence having no idea what they’re doing they end up someplace random in the space if you end up someplace random in the space it doesn’t care about us very much you know not literally random but random from our perspective we can’t predict it right okay um so that a thing that is smart enough to kill you knows that you’re there the concern is that it doesn’t care not that it cares about the same things you do and passes judgment on you for not separating your recycling but that it doesn’t care whether or not you separated your recycling and steps on you even though you did separate your recycling Liv I mean you were you were sort of saying that you were concerned about the way AI is going as well so what what does this AI catastrophe look like to you I mean that that’s that’s One Direction um there’s other ones such as well I mean again throughout history yes we are in an unprecedented territory so maybe history doesn’t apply but at the same time if we look back at history whenever a more powerful um and more intelligent group comes across a less powerful or intelligent group typically it didn’t work out well for the less intelligent group whether it was um Homo sapiens coming along the andols meeting the the homo sapiens uh or whether it was um Native Americans when when the they encountered the first uh Europeans didn’t work out too well for them either now maybe AI is going to have a higher morality than us and doesn’t care to do that or maybe it manages to unlock abundance and um not go and uh not need to SM us because we’re not in the way um maybe but there’s no guarantee of that we all we can do is look at history and go well that that doesn’t that didn’t work out so well so again like I’m I’m not saying I know how a catastrophe could play out I’m not saying I know that it definitely does I think there’s a chance that it can go well um all my my main point is that we are the way that we are going about it right now where it’s just about go as fast as possible optimize for like short-term metrics like maximizing shareholder value or getting the biggest hype who’s got the biggest funding round and so on is not the best way of ensuring that these really high stakes questions are answered the right way or at least this really high stakes shift this leveling up of essentially creating this new techn species because it’s not here like some people think it’s a purely a new technology purely a new tool I think that’s probably in a minority in this group um although I don’t want to speak for the others increasing numbers of people think it is it’s some kind of new species but even the word species is hard to apply because it’s got this like biological um lilt to it that it’s so it’s something so unbelievably new we just can’t possibly um predict where it’s going to go but as alaza pointed out the way way we’re going about it of all the possible things it could end up optimizing for it’s this massive search space um the number of things where it’s actually good for us or or at least where it can safely coexist with us compared to all the ways it can safely not coexist with us is much smaller um so it’s it’s I don’t I don’t know how it’s going to turn out I just know that the current way we’re going about it is horribly suboptimal okay but then you know let’s imagine for a moment it’s a tool right current it’s a technology as far as we know it’s not actually sent in conscious I mean we don’t know right so far I’m loving these really awkward faces that everyone is pulling but we’ve not seen any signs of it let’s just take it of face value right now we have these computer spreadsheets that seem to talk to us the way we like to be spoken to you can tell I’m not a computational scientist or anything like that but these things exist right have you used these systems yourself yes4 okay just check it yes of course I have of course I have I fall into the millennial category of always being polite to chat GPT because you never know if it you know if it becomes I don’t want it to think I don’t have manners and I was aware of that as I was using it for the first time and it’s clearly such a thing that Instagram feeds me memes about how that’s the case we are pointing at a very important Point Gish I think and this is the present systems that you’re looking at are not in the category of the systems that G and Ela that are live in are worried about when they talk about AI systems developing agency and displacing us the systems that we currently have are very different and while open Ai and entropic and others are making very big bets on the effect idea that these systems can be scaled to the point where they become superhumanly intelligent they’re not at the moment when you would take a system like Claude which is a very capable llm and you ask it about physics it’s not going to be nearly as good as Scott and if you ask it about decision theory is not going to be nearly as good as aaser right it’s uh the systems are not very good at making out of distribution uh predictions they’re not very good at creatively developing new thoughts making new reasoning testing their ideas with themselves learning from what they have done so far and while a lot of people are working on this we don’t yet know how well the present approach scales and maybe the secret is not in trying to imitate across many many people so you get to slightly superhuman performance per in many many human Dimensions um and basically approximated and some point suddenly some kind of breakthrough happens uh maybe the way is different maybe we need to get to systems that are able to reflect on their ideas and they need to reflect on uh how to get better that learn how to learn that uh develop themselves in very much the same way as we do to me in AI catastrophe is looks slightly different to me it means AI is not being built before Humanity Burns itself out suspect that this technological society that we are in is by itself not sustainable we are not able to keep it stable we are not able to survive within life on Earth if you continue with the present tools that we have and I think we need better tools to understand reality in ourselves and to me the big catastrophe is if we don’t achieve to build those tools and we have not a big history of making things better when we do them slower typically when we make them slower the California train is not going to be better than the trains in China just because we R them slow I’m also very vary of arguments that are being made against technological systems that would prevent from the internet to emerge right most of the arguments are being made to AI are things that apply even much much more to the internet itself all the dangers that has with dissemination of harmful information and so on but all this is dwarfed by the fact that by the benefits that the internet gives us imagine we should shut it down what would happen to us and it’s not just because became dependent on this dangerous evil technology I don’t remember anybody saying that the internet was going to be smarter than us and then wipe out Humanity like maybe somebody somewhere on Earth managed to say that but if so they were being silly the internet is not that kind of Technology the argument against AI is not that it’s going to spread misinformation it’s that it’s going to kill everyone there is a difference well I wouldn’t say that it’s silly are both AR these are both arguments that are made by different sets of people well so I mean on one hand we hear that you know this thing could kill all of us on the other hand we hear that our lives depend on it so let’s let’s take a few moments here to actually talk about whether or not AI could actually improve our human experience and if there are actually any potential benefits I mean is the only benefit is the only reason why there’s technology being developed right now so that companies can sell things better to us because they want to have these personalized mimics that will be tailored to my to encourage my consumption of things I mean surely there must be other things that it’s good for I mean there are massive and obvious potential benefits uh I I mean you know any any kid can can now you know use GPT to tutor them in any subject you know my daughter has used it a great effect seems very predictable at this point that the way that we will interact with our computers in a few years will be you know the Star Trek way that you know you just can tell it in English what you want it to do and it does that thing right uh uh you know that’s sort of but you know as predictable as it was in say 1993 that that yes you know the way that we interact act with each other will be via this worldwide web right it just you know you know it it simply hadn’t happened yet right but uh I think with existing llms if you just uh uh interface them with all of our other tools uh well enough then that could be done I mean drug Discovery I mean uh uh Alpha fold is already you know uh uh is starting to revolutionize you know the way that that uh biochemistry happens right so I mean I think the benefits uh uh on table are massive and obvious you know the question is whether we can get those benefits while avoiding catastrophes what surprises me is that like so many of these benefits that you’ve described don’t require AGI specifically they’re like just really really powerful narrow AIS um and I am fully supportive of those that’s I mean it’s fantastic provided again we think about potential externalities you know often when you unlock the ability to do drug Discovery you unlock the ability to create horrible uh novel chemicals that can just hurt humans um as we’ve already seen like that they just stuck a minor sign in front of it and uh develop 40,000 new scary chemicals that were like oh um so the bigger question as well is like is our are we already happy with the way our current system is set up because you could argue that as as you pointed out josa like that we’ve got so many highly complex problems that are careening us towards the cliff really fast you know we are extracting from our res you know from our environment faster than we can it can replenish um we’re very much in this like open loop style economy if we are just developing if if we’ve made a way to like to make AI suitably generalizable that every existing company can incorporate it to speed up what they’re already doing well now we’re just speeding up that the the the train that was already care kening off the tracks so it sounds like we’re we’re you’re hoping for a new a new form of sort of like Collective wisdom or something to emerge through AI that will like find better ways of realigning you know changing the direction of the track to something that is more sustainable and like like doesn’t just fall off the cliff is that correct I think it’s very much a possibility it’s uh it’s not a certainty also um not so much an a Optimist or uh a catastrophist I’m more an expectational I think that AI will at some point it’s quite unavoidable and we will need to prepare for being able to coexist with it and we should if we build it try to build it as well as we can okay and if we are preventing most responsible people from building AI it just means that the responsible people will stop building hi hi will not be built hold on hold on hold on sorry somebody needs to actually uh Define what an AGI is and how it’s different to the things that we talking about right now artificial general intelligence it’s able to do a very learn and therefore do a very broad variety of tasks much the way that humans are the most versatile animals on the face of the earth a bee builds beehives a beaver builds dams a human will look at both of those and envision a giant Dam with a honeycomb structure we learn many many different domains we do that much faster than animals evolve slightly different brains that can shift from hive to halves to dams so that’s the uniquely human capability that lets us walk on the moon despite having never evolved to breathe vacuum and an AI that ends up with the same thing is an artificial general intelligence and the big thing about llms from the science perspective is that they seem to although not be as broadly applicable as humans they’re trained using this one scheme then instead of being specialized to dozens and dozens of different tasks they seem to display some ability to pick up a variety although not a humanly wide variety of different tasks after they were trained a very short definition it’s basically a system that is in those areas where it’s mathematically possible to make useful models it can make those models and in a very narrow sense it’s a system that’s better than humans AI research because that’s the point that where we can do the rest of whatever that is and we can go to the beat I think I think that’s a attempted definition of artificial super intelligence rather than artificial general intelligence a gener I mean I mean by by some by some def by some reasonable sounding definitions of AGI you know the current language models would already be agis you know on the other hand we can point to things that they can’t do such as you know find a proof of the remon hypothesis or you know of a great or or answer a number of simple word problems that have been phrased in a slightly tricky way or for that matter you know like do correct Algebra derivations I recently asked chat pt4 to solve a physics problem and it just straightforwardly screwed up the algebra because as we all know AIS are bad at math but the reason why I asked this is because of Liv’s point right you sort of said well you know when I said are there any potential benefits everyone was like well of course of course there are and then Liv you was saying something along the lines of at least what I heard was that well certain specific AIS for specific tasks can be really helpful potentially could be potentially helpful we already have some of them and no one can deny that they’ve definitely Advanced us it was this AGI thing that was the thing that was of concern which is why I asked for those definitions so you know we’re here to talk about what artificial intelligence might do to humanity humankind and you know as we’re approaching the end of our time I I think it it’s that’s an important Nuance that you’re pointing out it’s to do with the sort of the form of artificial intelligence seems to be different depending on what it’s doing and what kinds of tasks that it’s capable of doing which presumably is to do with how we set them up and there’s some forms that we’re scared of basically right we’re saying they could kill us and some that we’re not scared of and so I guess most people who will be listening or thinking about the AI debate who aren’t Specialists are trying to make an assessment of you know is AI something worth being scared of or is it something that brings enough positives potentially into our lives to to entertain it enough to to enter that gray area so is the well I would say we’re probably in the gray area already and the question I put before you therefore is are we happy to remain in the gray area to tackle these questions of Consciousness where our boundaries lie what do we think reflect on our past or is this the situation where we feel we should Retreat Retreat immediately conceding all all potential benefits or is it actually worth facing those risks there a big spectrum of opinions you’re probably not able to resolve this completely people like elazer who think we should definitely Retreat because it’s too dangerous to step into this territory and there are a number of other people who think this is nonsense we will be able to control these Technologies because they develop relatively slowly and we are able to keep track of what we doing we have big progress and mechanistic interpretability we understand what these systems are doing better and better and can control how they are deployed and we don’t know what’s true in this space it’s uh we have to deal with this uncertainty and Poss can’t know what’s true because that implies that you don’t pass any regulations and they get to go on doing whatever they want you know the the answer ah we can’t possibly tell if this is dangerous or not is one that has been floated many Past Times in history with respect to Endeavors that in retrospect should have been shut down notably like cigarettes and the you know if you’re familiar with the history of like bad statistics being used to show that you couldn’t prove that cigarettes were causing lung cancer you know the answer that the claim that you can’t know something is also you know very politically questionable uh my my claim is not that like the potential benefits are small or that the potential dangers are very large it says we’re on course to not get the benefits and get the catastrophe you know we’re on course to to screw it up at the moment that’s that’s the CL case for backing off not the Poss but you’re making an outof distribution generalization that is uh not backed up by empirical evidence so far and uh well you can be correct it’s not sound first principles reasoning that has no Alternatives and basically your reasoning is flawed by the fact that you do not uh consider any alternatives to your particular reasoning anymore and uh I think this is a limitation live what what about you so I mean I I don’t I don’t describe myself as either a catastrophist or a pure Optimist I think it just it really depends how we go about this um I do think that the current incentive structures are not sufficiently aligned um with the long-term flourishing because as I described in my opening pitch um markets don’t perfectly select for um human well-being they select for creating as much appearance of human well-being uh such that they can sell their product and because that is the current dominating um method of developing this that that where as this technology gets more powerful that slight decoupling is going to become EXA at and you know there’s only so much variance we can take is my is my concern so that’s what that’s one thing my my other concern is more this this this thing I fled to josa which is like if our general economic system is already so misaligned and we are busy piling on AI to speed up that system in every single realm well that’s only going to surely ex exaggerate the misalignment so what I’m hoping for the thing that I’m like praying for I spend my life thinking about coordination problems I hope that proper like really Advanced AI is a thing that helps us solve coordination problems Fel finds ways to like better align incentives so that’s what I’m like really bullish on and I want to see more of that essentially more like I I don’t know like wisdom a wise AI whatever the that sorry whatever that looks like um is what we should be like really putting more effort into and I see very few people doing that there’s like like I don’t know there’s a couple of companies that are thinking about it but a lot of them are just like how do we ride the hype chain because it’s not even necessarily their fault they’re just in such a rat race there’s such few you know there’s a really scarce resource which is like intellectual capital and also VC funds to an extent although that seems to be increasingly less scar um they’re in this rat race of who can out compete the other and make each other dance and that means that these important like reverence questions and like questions of like how do we actually go about this with wisdom are going by the wayside so that’s my concern the case for catastrophe is not tied to some like narrow theory of exactly how minds work it’s that ignorance is not your friend it’s that not knowing which lottery numbers are drawn does not mean that you get to win the lottery when you have one or many slowly or quickly when you have super intelligent systems around things that are smarter than you if we have screwed up our entry into that we do not go back get to go back and try again because they don’t let us there’s a sense in which you’ve got to get it right on the first really serious try and to get well the first time it really matters and to get anything right when you have not tested it under similar circumstances before is a really really difficult ask in any field of engineering let alone one based on rather murky science where many of the fundamental principles are not understood the current systems are beh behaving in ways that seem cute and fun and because they don’t actually have the capability to hurt anyone but when they do misbehave like that you know it’s not that anyone can go into the system and figure out what happened and will it happen again the the malfunctions seem so fun and harmless because we have no idea what underlies them it’s like trying to put together a nuclear power plant or a nuclear weapon while having no concept of nuclear physics of course the first time it gets large enough it’s going to melt down that they’re they’re going in blind and even if they weren’t going in blind it would still be a very difficult ask to the first time we get lots of systems around that are super intelligence that could take us out that therefore for the first time have a motive to try to take us out to have them not take us out to steer the outcome we are not on course to get the benefits we are on course to get the catastrophe that is the case for backing off Scott I think that it is like overdetermined at this point that I I sort of will be centrally involved in everything that you know humans do uh uh in in this in this century and am I worried about uh a catastrophe destroying human civilization well of course I’m worried about that you know I was worried about that uh before uh recent advances in Ai and putting those things together now you know should I be worried about a catastrophe for civilization in which AI is centrally involved well you may notice that uh I have now committed myself to Yes uh but you know but but then there there’s a a crucial further question of what do we do about that now right given the you know immensity of of the of the uncertainties right I mean um um elzar has been talking about you know a a a a sort of r a random Choice from the space of possible Minds you know I don’t know how to think about that right he has a strong intuition that like a randomly chosen mind would be optimizing some goal that is very far from anything that we want um you know is that true um I don’t know it might be uh but uh uh you know I I think that that certainly something that we can do now is we can look at the powerful AIS that actually exist and that are being developed we can look at what harms they are causing or you know how PE bad s are misusing them to cause harm and we can try to mitigate those harms and we can hopefully learn something from that experience right you know I’ve been working at open AI on how do we recognize text that came from from from language models as opposed to text written by humans you know that can be used for lots of categories of misuse you know not just students cheating on their homework okay uh we can uh try to advance interpretability meaning understanding what is going on on inside these models um but you know one thing that’s that’s become very very clear to me you know for example from the uh the drama at open AI you know this past November uh about the the board and and Sam mman uh is that uh if there is going to be a a pause or a Slowdown you know the kind of thing that elezar has been calling for uh it is not going to happen based on someone just you know having bad vibes or some making an abstract argument that you know they don’t like the direction that things are going in uh it is going to have to be in response to well you know either some smaller scale catastrophe you know a uh a a a Chernobyl a you know Titanic sinking you know a a uh you know a thing that that uh you know or uh uh uh hopefully uh rather than reaching that point uh it you know there is some experiment in a closed environment that gets done that shows everyone here is what the danger is you see what could now happen and so I’ve become more and more convinced of the sort of central importance of uh uh you know getting uh uh clear warning shots you could call it you know gain of function research or uh uh um you know getting getting clear uh uh assessments of what these systems can do uh before they’re deployed because I think that if there were to be app pause uh uh uh that would be absolutely necessary to to to coordinating everyone around it as a voter I don’t think you so much I’m really really sorry we have to I have to call time on this as ever there’s never enough time and um I want to thank you all for taking part in in the last hour and and two minutes because I did let Scott run thank you again to all four of you for a very Lively debate I can see uh that this is not a simple subject and one that has a lot of potential but also a lot of considerations and I hope that we continue to have people like yourselves to to ensure that we as humans Stay Stay within the rails and help actually achieve our own potentials going forward thank you