Comment by Toby Ord

I guess one thing I could just say first is it’s not necessarily a 1-in-10 chance that we go extinct due to advanced AI; it was a 1-in-10 chance of existential catastrophe due to AI. I think it might have been Paul Christiano where I saw this laid out best, but it’s not clear in the real AI catastrophe scenarios whether everyone will die. In particular, if you were a superintelligent AI whose values were not aligned with humans, it could still be that humans are the most interesting thing to learn from on the planet, and that you would want to have some of them around in the future. Or maybe that you’d need them to service your server farms or something like that. But those would still be scenarios in which humanity’s potential is destroyed, and that we exist merely as slaves or servants to these AI systems that don’t value us in and of ourselves, and would protect themselves from any attempts we would make to destroy them. That’s to say the whole point of the idea of existential catastrophe is that there’s not much need to distinguish those two scenarios — they’re both like, we lose like 99% or more of the value that we could have had — and so it’s a perfect case for using the term.