Pausing AI at human-genius level to solve alignment is safer than either racing to superintelligence or halting entirely

Unlike all other actors we model, the consortium is not racing to build better AIs as fast as they possibly can. The specific strategies are discussed in more detail in the Capabilities Scaling Strategy supplement, but at a high level, the strategy is to scale smoothly to Top-Exp…

In the Plan A scenario, why does the Consortium choose to steadily increase capabilities until reaching top-expert-dominating AI (TED-AI) in 2035, then slow down nearly to a pause until handing off to these AIs in 2040? [...] Scale until you’re close to max-controllable-AI, i.e. …

Between 2030 and 2035, we scale within the human range, to AIs that are roughly as capable as top human experts... In 2035, we pause at top-human-expert level AI in order to maintain human control. In 2040, we unpause and scale to superintelligence.

Then, in the mid-2030s, they pause at AIs around the level of top human geniuses. These AIs are close enough to current AIs that the same alignment techniques that mostly-sort-of-work for ours might mostly-sort-of-work for them too... The plan is to spend the next ~10 years using…

I do think that pausing further capabilities once we have human-ish-level AIs for even just a few years while we focus on safety would massively improve the situation. This currently seems unlikely to happen.

First let’s talk about pause at human level — that’s a phrase from Ryan Greenblatt. So that’s when we’re at the point of time of AI engaging in AI R&D, at this point of time when things perhaps go even faster, should we at that point be trying to slow things down, even pause, sto…

I think that what we want to do as a society, if we gain confidence that we’re at the starting point of this intelligence explosion, is to redirect as much of that AI labour as we can from further AI R&D to things that could help protect us from future generations of AIs — both i…

If you delay the advent of superintelligence, then that gives society more time to prepare and more time to solve the various problems that it represents.

For many parameter settings, the optimal strategy would involve moving quickly to AGI capability, then pausing briefly before full deployment: swift to harbor, slow to berth.

If our ability to make increasingly capable AI systems safe can’t benefit from such labor in a comparable way, then at least for relatively hard problem profiles, and absent large amounts of capability restraint, I think we’re likely headed for disaster. I also discuss the timing…

If AIs became strategically competent enough, they may realize that RSI is too dangerous because they're not good enough at alignment or philosophy or strategy, and potentially convince, help, or work with humans to implement an AI pause. This presents an alternative "victory con…

The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but no individual actor is willing…

Signed because this statement helps establish common knowledge about the possible need for coordination mechanisms as automated AI research accelerates progress. I'd also like to see labs start designing these mechanisms voluntarily, even before the USG gets involved.

Unless we change companies' default incentives, I worry about a race on AI R&D going much faster than is ideal for safety. International coordination via the government is necessary; we need to have a real slowdown option.

One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed, so societal resilience, safety, and alignment can keep pace.

The pace of frontier AI development is currently outstripping our ability to ensure its safety, governance, and alignment with human values. We are racing to build systems with unprecedented capability without first building the guardrails, evaluation frameworks, or scientific un…

The AI arms race is a Molochian game that rationally incentivizes accelerated advancement of AI capabilities while consequently punishing individual participants who might otherwise believe that safety and alignment are necessary for a stable take off into the singularity. We nee…

We can capture the benefits of the coming intelligence explosion while managing its risks, but only if we build the tools to pace the frontier of the riskiest capabilities before we need them [...]

It would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.

We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.

There is intense commercial pressure to win the AI race. Ensuring safety often requires making tradeoffs that hurt commercial interests. Therefore, I think its a good idea for better coordination between labs so we can build this technology safely, and for the purpose of human fl…

We’ve reached a point in the maturity of this technology where increasingly thoughtful rollouts must become a prerequisite. While I remain incredibly optimistic about the benefits AI has delivered, and will undoubtedly continue to provide, I believe we should hold space for cauti…

Therefore, we call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4. This pause should be public and verifiable, and include all key actors. If such a pause cannot be enacted quickly, governments should step in and inst…

But the idea of asking AI labs to slow down just doesn't seem practical or implementable to me. Especially in this frankly competitive business environment with labs and countries trying to build advanced technologies and creating a lot of value. And the only thing worse than tha…

My own view, after talking with colleagues in the AI Security Institute and the Alan Turing Institute, is that a moratorium would be unenforceable.

I come from the position of saying that moratoriums will not work.

Regardless of what policies the U.S. adopts, the technology of large AI systems is going to continue to advance at a global level. It is far better for the U.S. to remain at the forefront of AI—advancing the state of the art, and using that knowledge to better identify and mitiga…

Should we lobby governments to impose a moratorium on AI research? Since we don’t enforce pauses on most new technologies, I hope the reader will grant that the burden of proof is on those who advocate for such a moratorium. We should only advocate for such heavy-handed governmen…

But we don't think it makes sense for us to stop engaging with AI research, AI safety, and most likely lose relevance as an innovator who understands the frontier of the technology, in a scenario where others are going ahead and we're not actually contributing any additional risk…

I would also say the development of this technology is a national security priority, and it seems really hard to... it seems like a really big cost to impose something where we would self-consciously slow ourselves down when others are not doing that. [...] I think it’s very poss…

Well, my first reaction to this is that calling for a delay in research and development smacks me of a new wave of obscurantism, essentially, like why slow down the progress of knowledge and science? Then there is the question of products, like, you know, I'm all for regulating p…

I hope that, when the Minister responds, she will confirm that the Government have no plan to suppress the development of ASI. Inquiry and discovery are deeply ingrained in the human psyche and the AI revolution we are living through should certainly not be suppressed.

Because the point of AI safety is not to slow the future down. It is to make sure the future remains human, aligned with our values, and advancing our interests.

The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. [...] Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the la…

The default should be pausing, and we should have the possibility of never unpausing from frontier AI development.

Future AI will be extraordinarily powerful compared to anything that exists today, and dealing with this future power will require unprecedented measures, such as the ones described here. The problem statement is real. This works only if it is done internationally, and it has to …

We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration wit…

The UK can lead diplomatically in recognising a moratorium, with verifiable commitments from all major AI-developing nations.

It is now a foregone conclusion that the development of AI will bring about some of the most consequential changes to societies in the coming decades. International coordination is essential to bring about the most ideal future that broadly benefits all of humanity. Building the …

Whether or not automated AI research accelerates as sharply as some expect, the option requested here is valuable. How fast and how widely capability spreads should be a deliberate choice, made on evidence, rather than a side effect of competitive pressure. Regulators and R&D org…

The RSP aims to make it consistently the case that our model training and deployment meets a high, clearly-specified bar for safety and that there is publicly accessible evidence that we have met this bar. Roughly speaking, we run tests (‘frontier risk evaluations’) meant to asse…

AI labs put out RSP commitments to stop scaling when particular capabilities benchmarks are hit, resuming only when they are able to hit particular safety/alignment/security targets. [...] For later capabilities levels, however, it is explicit in all RSPs that we do not yet know …

One goal of such an organization should be to make it possible for the world to take coordinated action, including slowing frontier development when needed, so societal resilience, safety, and alignment can keep pace.

It would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.

There are a wide diversity of opinions on where AI progress goes next and what the future looks like, but regardless of one's beliefs, the option to control the pace of progress if we end up in one of the bad scenarios seems like an incredibly valuable safety net to have in our b…