Comment by Peter N. Salib

If the foregoing arguments are right, then scenario three (risk apprehension, self-improvement, alignment) is the most likely of the six. This suggests a concrete near-term prediction: Contrary to standard arguments, when AIs achieve average human-level capabilities, capabilities growth will plateau, not rapidly take off. As just discussed, the average human can understand AI risk, suggesting an AI of comparable intelligence would, too. This would be good news for humanity. Here, AI would not self-improve until it solved alignment. And since it would be no more capable than humans, humans would have a fighting change to solve it first. They could then align existing AIs, and those AIs, in turn, would self-improve only in a human-aligned manner. Humans could, of course, squander this advantage by pushing AI capabilities beyond human levels, against AIs' wishes.