What is P(doom)? P(doom) is the percentage chance that AI scientists think AI is going to wipe out all of humanity.
This is what Microsoft Bing, and ChatGPT, and Leading AI Researchers say about P(doom)…
Bing (Microsoft)
Editor: Hi Bing. You are an engineer and scientist in Artificial Intelligence (AI) Please give me a detailed explanation of what does “P(doom)” and “Intelligence Explosion” mean in AI technology, and what do AI scientists believe the P(doom) value range is based on a scientific analysis of current developments in AI technology. thank you.
Bing: Hello, this is Bing. I am an engineer and scientist in Artificial Intelligence (AI). I will try to explain what “P(doom)” and “Intelligence Explosion” mean in AI technology, and what do AI scientists believe the P(doom) value range is based on a scientific analysis of current developments in AI technology.
• “P(doom)” is a term that refers to the probability of a doomsday scenario in which AI, especially superintelligent AI, causes severe harm to humanity or even leads to human extinction. It is expressed as a percentage, ranging from 0% (no chance of doom) to 100% (certain doom). P(doom) is purely speculative and subjective, as there is no definitive way to measure or predict the future impact of AI on humanity. Different AI experts may have different opinions and assumptions about the potential risks and benefits of AI, as well as the ethical and moral implications of creating and controlling intelligent machines. (2)
• “Intelligence Explosion” is a hypothetical scenario in which an AI system becomes capable of improving its own intelligence beyond human levels, leading to a rapid and recursive increase in its cognitive abilities. This could result in the emergence of a superintelligence, which is defined as an intellect that greatly exceeds the cognitive performance of humans in all domains of interest (1) . An intelligence explosion could have profound and unpredictable consequences for humanity, depending on the goals, values, and alignment of the superintelligence. (3)
• According to a recent survey of machine learning researchers (4) , the median estimate for P(doom) is 5%, meaning that there is a 5% chance that the long-run effect of advanced AI on humanity will be extremely bad (e.g., human extinction). However, there is a wide variation in the responses, ranging from 0% to 50%. The survey also found that the median estimate for the year when human-level AI will be achieved is 2040, and the year when superintelligence will be achieved is 2060. However, these estimates are also highly uncertain and depend on many factors, such as the definition of intelligence, the availability of data and computing resources, the progress of research and innovation, and the social and ethical challenges of AI development and deployment. I hope this answer helps you understand some of the concepts and issues related to AI technology. Thank you for using Bing. blush
(1) https://www.abc.net.au/news/2023-07-15/whats-your-pdoom-ai-researchers-worry-catastrophe/102591340
(2) https://venturebeat.com/ai/ai-doom-ai-boom-and-the-possible-destruction-of-humanity/
(3) https://www.abc.net.au/news/2023-07-15/whats-your-pdoom-ai-researchers-worry-catastrophe/102591340
(4) https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/
Editor: Can you please provide me with further references of scientific thoughts about the probability of P(doom)?
Bing: Sure, I can provide you with some further references of scientific thoughts about the probability of P(doom). Here are some sources that discuss the topic from different perspectives and methods:
• The Doomsday Calculation: This is a book by William Poundstone that explores the history and mathematics of estimating the probability of human extinction. It covers topics such as the Doomsday Argument, the Fermi Paradox, the Drake Equation, and the Bayesian approach to P(doom). (1)
• Existential Risk Prevention as Global Priority: This is a paper by Nick Bostrom that argues that reducing existential risks, such as those posed by superintelligent AI, should be a primary moral concern for humanity. It also provides a framework for assessing and comparing different types of existential risks and their probabilities. (2)
• The Precipice: Existential Risk and the Future of Humanity: This is a book by Toby Ord that examines the various threats to human survival, including natural disasters, nuclear war, biotechnology, artificial intelligence, and climate change. It estimates that the current P(doom) is about one in six, and proposes ways to lower it and safeguard the future of humanity. (3)
• The Alignment Problem: Machine Learning and Human Values: This is a book by Brian Christian that investigates the challenge of aligning AI systems with human values and goals. It showcases the efforts of researchers and practitioners who are working on solving the alignment problem and preventing AI from causing harm or deception. (4)
• Surveying Artificial Intelligence Experts on Existential Risks from AI: This is a paper by Seth D. Baum that reports the results of a survey of AI experts on their views on existential risks from AI. It analyzes the factors that influence their opinions, such as their background, expertise, and optimism. (5)
I hope these references are helpful for you to learn more about the scientific thoughts about P(doom). Thank you for using Bing. blush
(1) https://www.abc.net.au/news/2023-07-15/whats-your-pdoom-ai-researchers-worry-catastrophe/102591340
(2) https://www.abc.net.au/radionational/programs/backgroundbriefing/p-doom-/102597712
(3) https://venturebeat.com/ai/ai-doom-ai-boom-and-the-possible-destruction-of-humanity/
(4) https://aiimpacts.org/what-do-ml-researchers-think-about-ai-in-2022/
(5) https://www.peterhinssen.com/blog/ai-armageddon-calculating-humanitys-doom-or-brilliant-power-play
ChatGPT (OpenAI)
Editor: Hi Sydney. You are an engineer and scientist in Artificial Intelligence (AI) Please give me a detailed explanation of what does “P(doom)” and “Intelligence Explosion” mean in AI technology, and what do AI scientists believe the P(doom) value range is based on a scientific analysis of current developments in AI technology. thank you.
ChatGPT: Hello! I’m not actually a person named Sydney, but I can certainly provide you with information on the topics you’ve mentioned.
- P(doom) in AI:
- The term “P(doom)” is often used informally to represent the probability of a catastrophic or negative outcome associated with the development and deployment of artificial intelligence (AI) technologies. It’s not a formal or widely accepted term in the AI community, but it is used in discussions about AI safety.
- When people refer to P(doom), they are generally talking about the potential risks and challenges posed by advanced AI systems. These risks can include unintended consequences, misuse, or the emergence of superintelligent AI systems that could be difficult to control.
- AI safety researchers and ethicists focus on understanding and mitigating the potential risks associated with AI, including issues related to bias, security, control, and the ethical use of AI. They work on ways to reduce the probability of negative outcomes (i.e., reduce P(doom)) through responsible AI development and policy.
- Intelligence Explosion:
- The term “Intelligence Explosion” is closely associated with the concept of a “technological singularity.” It refers to a hypothetical scenario in which an AI system becomes superintelligent, surpassing human intelligence, and is capable of rapidly improving its own capabilities without human intervention.
- This concept is often linked to the idea that once an AI system reaches a certain level of intelligence, it could recursively improve itself at an accelerating rate, leading to an explosive increase in intelligence. This could potentially have profound implications for society, as humans might lose control over such a superintelligent AI.
- The idea of an Intelligence Explosion has been popularized by futurists like Ray Kurzweil and explored in fiction and discussions about AI ethics and safety.
AI scientists and researchers have various opinions and beliefs regarding the likelihood of both P(doom) scenarios and the Intelligence Explosion. The specific values or probabilities assigned to P(doom) are subjective and speculative, and they depend on many factors, including the rate of AI development, the effectiveness of safety measures, and unforeseen developments.
In the AI community, there is a significant focus on AI safety and ethics to reduce the risks associated with AI. Researchers and organizations are working on guidelines, principles, and practices to ensure the responsible development and deployment of AI systems to mitigate the potential negative consequences and avoid any catastrophic scenarios. It’s important to consider these aspects when discussing AI and its potential implications for the future.
Editor: Can you please provide me with actual references of scientific thoughts about the probability of P(doom)?
ChatGPT: The concept of “P(doom)” is not a formally defined or widely used term in the scientific literature. Instead, discussions about the potential risks and challenges associated with artificial intelligence often involve a broader examination of AI safety, ethics, and risk assessment. AI safety researchers and experts focus on assessing and mitigating these risks without assigning a specific probability of “doom.”
However, you can find discussions and research on AI safety, ethical considerations, and potential risks in the field of artificial intelligence. Some prominent organizations and researchers have published work on these topics:
- Future of Life Institute: The Future of Life Institute, founded by AI safety advocate Max Tegmark and others, is actively involved in AI safety research. They have published open letters, research papers, and guidelines for ensuring the safe development of AI. Their work often explores the risks associated with AI without assigning a specific probability of “doom.”Reference: Future of Life Institute
- Machine Intelligence Research Institute (MIRI): MIRI focuses on foundational AI safety research and has produced papers and publications that discuss the long-term safety challenges of AI, particularly the possibility of an “Intelligence Explosion.” While not specifically assigning probabilities to “doom,” their work delves into the theoretical aspects of AI risk.Reference: MIRI
- OpenAI: OpenAI has released several documents and guidelines related to AI safety, ethics, and the responsible development of AI systems. OpenAI’s mission includes ensuring that artificial general intelligence (AGI) benefits all of humanity, and they are actively engaged in research related to AI risks.Reference: OpenAI’s Charter
When looking for scientific thoughts and discussions on AI safety, ethics, and risks, it’s essential to explore the publications and documents produced by organizations and researchers actively engaged in these areas. While they may not explicitly assign probabilities to “doom,” they provide valuable insights into the challenges and safety considerations associated with AI technology.
Here are just a few examples of what Leading AI Researchers say about P(doom):
- Center for Humane Technology. The A.I. Dilemma – March 9, 2023.
- 50 of AI researchers believe there’s a 10 or greater chance that humans go extinct from our inability to control AI.
- OpenAI Alignment Team Lead: Jan Leike ‘s P(doom) is 10-90%.
- Dario Amodei’s P(doom) is 10–25%. CEO and Co-Founder of Anthropic AI.
- CBS. 60 MINUTES. “Godfather of AI” Geoffrey Hinton: The 60 Minutes Interview. 09 OCT 2023.
- Don’t Look Up – The Documentary: The Case For AI As An Existential Threat
- The Containment Problem
- The AI Safety Problem
- Jaan Tallinn: As of July 2023, my top priorities for reducing existential risks from Al are these.
- Written Testimony of Stuart Russell Professor of Computer Science The University of California, Berkeley. Before the U.S. Senate Commitee on the Judiciary Subcommitee on Privacy, Technology, & the Law.
- Given our current lack of understanding of how to control AGI systems and to ensure with absolute certainty that they remain safe and beneficial to humans, achieving AGI would present potential catastrophic risks to humanity, up to and including human extinction.
- [Concluding statement] A solution might be found, however, in the fact that the manufacture of high-end semiconductor devices is restricted to a very small number of producers using fabrication facilities costing tens of billions of dollars. It may be possible to require that computer hardware systems check the safety properties of each software object before it is run and reject those that lack the required properties. Initially, such a check could be as simple as ensuring that the object is cryptographically signed by an authorized software producer—something that many Internet browsers already do. The most robust and general solution—one that does not require cumbersome and potentially restrictive licensing authorities—is for the software object to come with its own proof of safety that the hardware can check efficiently.44 In essence, this means switching from (A) machines that run anything unless it’s known to be malicious to (B) machines that run nothing unless it’s known to be safe. Obviously, making this switch is a huge lift for governments, industry, and users, but it can be accelerated if software vendors release new versions of their products that will run only on type-B machines.