ROMAN: The problem of controlling AGI or super intelligence in my opinion is like a problem of creating a Perpetual safety Machine by analogy with perpetual motion machine it’s impossible. Yeah we may succeed and do a good job with GPT five, six, seven, but they just keep improving learning eventually self-modifying interacting with the environment interacting with malevolent actors. The difference between cyber security narrow AI safety and safety for General AI for super intelligence is that we don’t get a second chance. With cyber security somebody hacks your account what’s the big deal you get a new password new credit card you move one. Here if we’re talking about existential risks you only get one chance. So you really asking me, what are the chances that will create the most complex software ever, on a first try, with zero bugs and it will continue have zero bugs for 100 years or more.
LEX: So there is an incremental Improvement of systems leading up to AGI to you it doesn’t matter if we can keep those safe there’s going to be one level of system at which you cannot possibly control.
ROMAN: I don’t think we so far have made any system safe at the level of capability they display. They already have made mistakes. We had accidents. They’ve been jail broken. I don’t think there is a single large language model today which no one was successful at making it do something developers didn’t intend it to do.