I. J. Good speculated, 58 years ago, in 1965 that artificial general intelligence might bring about an intelligence explosion:

• ‘Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.’

The story-of-the-century for all humanity?

The spontaneous and uncontrolled emergence of AGI. 

(incidentally, it could be the last story ever told by humans)

Who believes AGI could be an existential threat to humanity?

• Yuval Harari, Tristan Harris and Aza Raskin – We have summoned an alien intelligence. We don’t know much about it, except that it is extremely powerful and offers us bedazzling gifts but could also hack the foundations of our civilization. – The New York Times. March 24, 2023
  
• Elon Musk– I think we should be very careful about artificial intelligence. If I were to guess like what our biggest existential threat is, it’s probably that. So we need to be very careful with the artificial intelligence. Increasingly scientists think there should be some regulatory oversight maybe at the national and international level, just to make sure that we don’t do something very foolish.With artificial intelligence we are summoning the demon.’ – The Washington Post, October 24, 2014 – “I Tried To Warn You” (2023)
• Nick Bostrom. What happens when our computers get smarter than we are? TED (2015)
• Stuart Russell – Provably Beneficial AI (2022)
  
• Max Tegmark  – I think it’s pretty clear that artificial intelligence and the quest for ever more intelligent non-biological minds is going to ultimately become the best thing ever to happen to humanity or the worst thing ever to happen to humanity. (2022)
  
• Thomas Friedman – Our new Promethian moment. 21 March 2023
  
• Ezra Klein – This Changes Everything. 14 March 2023
• Future of Humanity Institute (FHI), University of Oxford
• UC Berkeley Center for Human-Compatible AI (CHAI)
• Future of Life Institute
• Machine Intelligence Research Institute Berkeley, California
• Deepmind, Alphabet Inc.
• Salesforce AI Research – Trusted AI
• Center for AI and Digital Policy
• The Council of Europe AI Strategy
• UNESCO. Ethics of Artificial Intelligence
• Anthropic
• Microsoft
• Artificial intelligence films

In 2022, over 700 top academics and researchers behind the leading artificial intelligence companies were asked in a survey about future A.I. risk. Half of those surveyed stated that there was a 10 percent or greater chance of human extinction (or similarly permanent and severe disempowerment) from future A.I. systems.

• Bill Gates – The age of AI has begun. 21 March 2023