Comment by Nora Belrose

Should we lobby governments to impose a moratorium on AI research? Since we don’t enforce pauses on most new technologies, I hope the reader will grant that the burden of proof is on those who advocate for such a moratorium. We should only advocate for such heavy-handed government action if it’s clear that the benefits of doing so would significantly outweigh the costs. In this essay, I’ll argue an AI pause would increase the risk of catastrophically bad outcomes, in at least three different ways: reducing the quality of AI alignment research by forcing researchers to exclusively test ideas on models like GPT-4 or weaker; increasing the chance of a “fast takeoff” in which one or a handful of AIs rapidly and discontinuously become more capable, concentrating immense power in their hands; and pushing capabilities research underground, and to countries with looser regulations and safety requirements.