Full question: "An AI system will cause X thousand deaths, or X billion dollars in damage, within a 3-day period

Resolution Criteria:

"Will cause" means that the AI system was the primary cause of the deaths or damage. This could be thought of as who is legally (or morally) at fault. For example, if self-driving cars become commonplace, the total number of at-fault deaths caused by self-driving cars would count towards this total. Or, if an AI system designs a bridge (and no human reviews the design), then a collapse of that bridge would count towards this total. But if a human engineer works with a (known-to-be-fallible) AI system, and reviews and signs off on the designs, then any failures would be the fault of the human.

Motivation and Context:

Current AI systems are not generally put in control of any situation where they could cause catastrophic damage. But this may change. If we are to trust AI systems with more and more important decisions, we need to know how much damage they could cause if they fail. Others have a more existential worry: that AI systems could actually intentionally cause mass destruction via "misalignment". I want to know how much damage the most severe AI catastrophe will be in the next few years.

Question copied from: https://nicholas.carlini.com/writing/2024/forecasting-ai-future.html