Comment by Joe Carlsmith

That is, I think a wiser and more coordinated civilization would likely be employing quite a lot of capability restraint in building advanced AI, especially as we start to approach transformatively powerful systems – and this despite the potential costs of delaying the benefits of safe superintelligence both to present-day people and to our civilization’s broader existential security. [...] Overall, and without trying to work through all the possible abstract dynamics here comprehensively, my current best guess is that in a world of idealized coordination and decision-making around loss of control risk – one where fully effective slow-downs could be initiated and then unblocked effectively, rationally, and without significant side effects beyond those implied by the slow-down of the technology itself – then we would indeed see significant (read: multi-year) restraint on the development of artificial superintelligence while we improve our understanding of how to ensure its safety.