Pausing AI at human-genius level to solve alignment is safer than either racing to superintelligence or halting entirely

In the Plan A scenario, why does the Consortium choose to steadily increase capabilities until reaching top-expert-dominating AI (TED-AI) in 2035, then slow down nearly to a pause until handing off to these AIs in 2040? [...] Scale until you’re close to max-controllable-AI, i.e. …

Then, in the mid-2030s, they pause at AIs around the level of top human geniuses. These AIs are close enough to current AIs that the same alignment techniques that mostly-sort-of-work for ours might mostly-sort-of-work for them too... The plan is to spend the next ~10 years using…

If our ability to make increasingly capable AI systems safe can’t benefit from such labor in a comparable way, then at least for relatively hard problem profiles, and absent large amounts of capability restraint, I think we’re likely headed for disaster. I also discuss the timing…

I think that what we want to do as a society, if we gain confidence that we’re at the starting point of this intelligence explosion, is to redirect as much of that AI labour as we can from further AI R&D to things that could help protect us from future generations of AIs — both i…

First let’s talk about pause at human level — that’s a phrase from Ryan Greenblatt. So that’s when we’re at the point of time of AI engaging in AI R&D, at this point of time when things perhaps go even faster, should we at that point be trying to slow things down, even pause, sto…

I do think that pausing further capabilities once we have human-ish-level AIs for even just a few years while we focus on safety would massively improve the situation. This currently seems unlikely to happen.

The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing…

A baby step would be some sort of agreement between OpenAI and Anthropic as the leading Western labs. [They would need] some sort of neutral understanding that they won't immediately go into recursive self improvement in the next year.

As someone who works on recursive self-improvement and automated evaluation, I sign this because the diagnosis is right and the ask is refreshingly humble: pace, not pause. The phrase that matters is "automating AI research": that's the point where systems start improving themsel…

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

Signed because this statement helps establish common knowledge about the possible need for coordination mechanisms as automated AI research accelerates progress. I'd also like to see labs start designing these mechanisms voluntarily, even before the USG gets involved.

We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels.

One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed, so societal resilience, safety, and alignment can keep pace.

We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.

Responsible AI development requires more than capable models. It requires independent monitoring, containment, clear limits on authority, and mechanisms for coordinated pacing if automated research begins to outrun those controls. Building that capacity now is not opposition to p…

We should take decisive action now so that we can slow down, as the CEOs have recently acknowledged, slow down the pace of development in order to protect the American people and ensure that AI is proceeding safely.

We should rapidly develop powerful AI tools that help cure diseases and solve practical problems, but not autonomous smarter-than-human machines that nobody knows how to control. Creating superintelligent machines is not only unacceptably dangerous and immoral, but also completel…

Superintelligence would likely break the very operating system of human civilization – and is completely unnecessary. If we instead focus on building controllable AI tools to help real people today, we can far more reliably and safely realize AI’s incredible benefits.

If we race to build superintelligence without clear and morally informed parameters, we risk undermining the incredible potential AI has to alleviate suffering and enable flourishing. We should intentionally harness this amazing technology to help people, not rush to build machin…

To get the most from what AI has to offer mankind, there is simply no need to reach for the unknowable and highly risky goal of superintelligence, which is by far a frontier too far. By definition this would result in a power that we could neither understand nor control.

But the idea of asking AI labs to slow down just doesn't seem practical or implementable to me. Especially in this frankly competitive business environment with labs and countries trying to build advanced technologies and creating a lot of value. And the only thing worse than tha…

Regardless of what policies the U.S. adopts, the technology of large AI systems is going to continue to advance at a global level. It is far better for the U.S. to remain at the forefront of AI—advancing the state of the art, and using that knowledge to better identify and mitiga…

My own view, after talking with colleagues in the AI Security Institute and the Alan Turing Institute, is that a moratorium would be unenforceable.

For me, a one-sided moratorium which ties our hands behind our backs while the likes of North Korea and the al-Qaedas of the world crack on: no, I am afraid that that worries me even more.

I would also say the development of this technology is a national security priority, and it seems really hard to... it seems like a really big cost to impose something where we would self-consciously slow ourselves down when others are not doing that. [...] I think it’s very poss…

But we didn’t pause development and permanently mandate that they couldn’t go faster than horses. Eventually—after trial and error—we developed better and safer cars, as well as road safety laws to prevent accidents, and legal provisions like driver’s licences to ensure a minimum…

Well, my first reaction to this is that calling for a delay in research and development smacks me of a new wave of obscurantism, essentially, like why slow down the progress of knowledge and science? Then there is the question of products, like, you know, I'm all for regulating p…

Because the point of AI safety is not to slow the future down. It is to make sure the future remains human, aligned with our values, and advancing our interests.

Big AI companies should be allowed to build AI as fast and aggressively as they can [...] Startup AI companies should be allowed to build AI as fast and aggressively as they can. [...] Open source AI should be allowed to freely proliferate and compete with both big AI companies a…

Something that worries me about the general push towards a pause on AI frontier progress is that there is not a very clear thesis about what to do with extra time.

But we don't think it makes sense for us to stop engaging with AI research, AI safety, and most likely lose relevance as an innovator who understands the frontier of the technology, in a scenario where others are going ahead and we're not actually contributing any additional risk…

Should we lobby governments to impose a moratorium on AI research? Since we don’t enforce pauses on most new technologies, I hope the reader will grant that the burden of proof is on those who advocate for such a moratorium. We should only advocate for such heavy-handed governmen…

The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. [...] Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the la…

The default should be pausing, and we should have the possibility of never unpausing from frontier AI development.

Now. My take is that if we were to carefully analyse this question of exactly when we should slow down, it would be a while ago in the past, because we’re just too close to this crazy future. Trying to be super precise about exactly when in the future, like, no, no, no: “Already”…

I’m confident that a cooperative international approach (Plan A) is better than the alternatives because the alternatives involve being in a race during the intelligence explosion, which incurs huge risks. I’m not confident about the exact variant of Plan A, or whether Plan A as …

Future AI will be extraordinarily powerful compared to anything that exists today, and dealing with this future power will require unprecedented measures, such as the ones described here. The problem statement is real. This works only if it is done internationally, and it has to …

The question that the noble Lord posed was: do we therefore go for a moratorium? That would be highly desirable, but I do not think that it will be possible in the short term.

A full pacing, or even “pause”, in which participating governments agree to substantially limit the overall rate of AI development. I support floating this, but I think it is unlikely to actually happen any time soon.