Comment by Joe Carlsmith

But even in settings that assume significant idealization, the fact that pauses can incur ongoing background existential risk means that very high absolute standards here can lead us astray: the decision to pause or unpause needs to remain attentive ...

Safety progress takes time. If you don’t have enough time, at each stage of AI development, to ensure the AIs that you build in the next stage won’t destroy humanity, then you’ll fail, and humanity will be destroyed.

That is, I think a wiser and more coordinated civilization would likely be employing quite a lot of capability restraint in building advanced AI, especially as we start to approach transformatively powerful systems.