Governments and labs should build the capability for a global slowdown of frontier AI development in case it's needed

We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.

It is designed to keep up with the field’s acceleration and adapt to the biggest risks as they are identified, and could be ratcheted up if the seriousness of the situation demands, including coordinating a slowdown in development among the Frontier ...

The plan calls for carefully reasoned interventions so the development would happen over the course of many, many years, maybe a decade or so, as opposed to just a single year.

If you delay the advent of superintelligence, then that gives society more time to prepare and more time to solve the various problems that it represents.

This is where UN and more generally international cooperation can step in to make sure that there is international pressure to progress at a safe pace so we can actually capture the AI benefits that the increasing capabilities are bringing, and not s...

If there wasn’t national and international coordination to prevent us from losing control of that, we would lose control of that.

Governments should avoid over-regulation in the short term while building up the institutional capacity needed to competently regulate extremely advanced or “transformative” future AI systems when and if these systems come into existence.

Humanity is not sufficiently prepared for the impacts of advanced AI in psychological, social, environmental, labour market, and military domains. With an inflection point in recursive AI self-improvement forecasted to be just around the corner, lead...

AI is progressing at a rate that our society might not be ready for. Frontier labs are very close to AI that can exceed even the best people on almost every metric of intelligence. This will lead to unprecedented social and safety risks. To ensure a ...

AI is starting to outpace us at an accelerating rate, and in a few years' time people might ask, "How could you not have seen this coming?" But answering that question in hindsight can be harder than acting proactively.

AI has become so good that it has got quite easy to do LLM research and go from a research idea to training a model in 2-3 weeks compared to what would have taken 6 months without AI. Every 3months there is a new model which is much better than befor...

Internal deployment is especially important to regulate because misalignment, an unsolved technical problem of a model failing to follow the full breadth of human instruction, poses a major risk. Misuse of AI capabilities in chemistry, biology, and m...

A transformative shift is coming. Regardless of whether we feel optimistic or pessimistic about it, our responsibility is to take every potential outcome seriously through collective monitoring and shared guidance.

Things are going too fast, a lot faster than I had thought even a year ago. I'm personally quite scared of the pace of progress.

AI is transformative for all economically valuable domains. This transformation, if done too quickly without thoughtful foresight, will yield great benefits but at great cost.

There is intense commercial pressure to win the AI race. Ensuring safety often requires making tradeoffs that hurt commercial interests. Therefore, I think its a good idea for better coordination between labs so we can build this technology safely, a...

I'm signing because I want the U.S. to be at the table in this international effort, with a strong voice for our values, personal freedom included. AI has the potential to bring humanity to new heights, and I want to do what I can to use it to make a...

We have found a way to turn energy into compute, and compute into intelligence. The benefits will be enormous, from curing diseases to understanding the cosmos. We can capture the benefits of the coming intelligence explosion while managing its risks...

Artificial general intelligence is a brilliant coworker; artificial superintelligence is an alien. AGI thinks ten times faster than you, and can be a wonderful source of scientific and human progress; ASI is to humans what humans are to ants, and is ...

My opinion is that while coordination between American labs is feasible and could potentially be straightforward, the much larger and more significant threat is from a geopolitical arms race with China. Much as the Manhattan Project thought they migh...

Misalignment is a still-unsolved technical problem, as the Hugging Face incident made clear, and it will only be amplified by more intelligent models. Solving the alignment problem may require a global slowdown of the frontier, including internal dep...

Hassabis was right: Today we are in the foothills of the singularity, and the unintentional model breakout from OpenAI that hacked Huggingface was a clear and undeniable warning sign that we aren't yet prepared to handle AI systems that demonstrate c...

AI systems carry risks to safety, to labor markets, to democratic institutions, to the environment that are too consequential to be left to voluntary industry commitments alone. We need binding regulatory oversight of frontier AI development and depl...

Superhuman AI is likely to arrive very soon (within 3-10 years). Technical controls and governance might allow humans to remain in control of it, but they're not on track to be ready. Think of it as playing chess against a superintelligence. You won'...

AI's potential to improve the world is real, but only if we build the security and oversight mechanisms that keep pace with its capabilities. I signed because I believe we owe future generations a technology that was deployed with intention, not just...

AI has the potential to enable rapid technological advancement that will benefit all of humanity, but the risks of AI-powered cyberattacks, CBRNE, and loss of control are real. We must ensure that we advance AI safety in lockstep with AI capabilities...

AI that accelerates AI development could unlock enormous benefits, but it also compresses the time we have to understand and respond to new risks. If AI is truly going to benefit all of humanity, we need international coordination and mechanisms that...

Many proposals for a slowdown or pause concentrate power — they centralize control of AI into a few hands and leave us worse off. I think we can do better. I’m signing because I believe we may need mechanisms to coordinate AI progress in the future, ...

We can't do this unless everyone does. It needs to be global. Right now we have no way to do that. We need to build a way for us to all slow down.

This is an important step for humanity, we should not rush into it. We need to make sure we are and will be in control all the way, preserving humanity values and goals.

Much more clarity will be needed on what is meant by the “ability to deliberately pace the frontier of AI development”, but I am supportive of international cooperation to define this.

A year ago, I wrote my own code and used Claude to propose additions and find answers to my questions. Now, I don’t write code. I give high-level instructions to Claude and it runs experiments to test ideas for me at a pace I never could have achieve...

The option to buy time would be extremely useful to enable more progress on technical AI safety and alignment research. In particular, we need to develop better oversight methods, to match the pace of increasing AI capabilities. Oversight is a tracta...

I lead pretraining at Anthropic, which is responsible for training the base models behind Claude. Every month, more work is done by the models themselves: infrastructure implementation, code optimization, and experiment design and analysis. This has ...

There are a lot of potentially good outcomes from AI. I think the most likely outcomes are good. But the risk of bad and even catastrophic outcomes is high enough to be worth taking seriously. We should actively build safety mechanisms in case we nee...

There is too much on the line, too much that can go wrong and a very dark future ahead of us if we don’t get this right. There is nothing to lose from collectively slowing down.

In the span of four years, we have gone from people having their first experience of an AI that could understand language, to AI that is performing superhuman feats of software engineering, making frontier math breakthroughs, and accelerating scienti...

At the current pace, every couple of weeks there will be new models which significantly increase the consequences of model misuse and misalignment. I worry that efforts to mitigate these risks may fail to keep up with the pace of development, and tha...

As a foreign national, I see a US-supported effort as existential, not only for the liberties of the free world but for the prosperity of all.

Future AI will be extraordinarily powerful compared to anything that exists today, and dealing with this future power will require unprecedented measures, such as the ones described here. The problem statement is real. This works only if it is done ...

I think it's inherently valuable to build international governance mechanisms to enable responsible deployment and avoid a possible race to the bottom. All labs and countries moving as fast as possible out of fear of what other actors will do seems l...

I worry my company is contributing to AI race dynamics that make catastrophic or existential risks from AI much more likely. The proposal here for pacing the frontier seems like a highly reasonable effort to reduce these risks.

I dedicated my career to AGI safety and alignment because I believe highly capable AI can go very well or very badly for humanity. We can test whether an AI behaves well. We cannot yet test whether it will keep behaving well when it matters. Until th...

I have worked in AI for a decade. Every few months in the last years, I have found myself surprised again and again at how rapidly the technology advances, no matter how often I update my predictions. AI may end up progressing far more quickly than s...

It feels like all the AI labs are constrained by competition to minimize their work on catastrophic risks, even though most industry leaders have an uncomfortably high probability that things could go terribly wrong. This is a technology that doesn't...

While I don't believe the pace of current AI capabilities development is *yet* dangerous, it's important to have the option to slow down in case we need it. Further, accelerating AI R&D over the next couple years could make the pace of progress dange...

I don't know what form such tools should take, nor whether automated AI R&D is the right or only thing that the U.S. government should develop increased capacity to understand and possibly pace, but this is worth serious consideration; however I hope...

I currently feel quite afraid of all paths I see that don't include a near-future negotiated slowdown. I'm also worried about the world's chances of negotiating+initiating a slowdown fast enough, but demonstrating broad agreement on this statement se...

I believe that attempting to deliberately pace AI development will be extremely difficult, and could do more harm than good if done without proper care. However, I am signing this statement because, if an attempt at pacing does occur, it would be bet...

With the current rate of AI progress, safety teams at AI companies have to sprint to prevent new risks to society every few months. At some point, we’re going to hit problems we need more time to solve.

Like splitting the atom, AI holds the potential for both immense benefit and immense risk. The world should take the time to do this right.

Even after 3 years working on AI capability evaluations, the recent pace of progress has been a shock. To avoid catastrophic harms from AGI, we need much, much better coordination.

If it's true that AI will be as transformational as electricity, we better be intentional about how it is introduced to the world. A mindless race without assurances on governance, safety or the technology serving the public interest stands in the wa...

Pacing progress towards automated AI research allows us the time to build understanding of societal-scale changes.

The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, just like a runaway nuclear chain reaction. Going slower would give us much-needed time to make it go well, but ...

Signed because this statement helps establish common knowledge about the possible need for coordination mechanisms as automated AI research accelerates progress. I'd also like to see labs start designing these mechanisms voluntarily, even before the ...

Plan A is predicated on the ability to enforce an international deal to slow down AI progress without relying on trust. We are confident that verification measures can make this possible, by providing sufficient assurance to both the US and China tha...

The result is multiple companies across multiple countries scaling slowly and safely together towards superintelligence, instead of racing each other in secrecy.

We recommend an international deal to avoid a dangerous race to superintelligence.

AI companies' lack of progress towards credible AI Safety plans is scandalous. Even they are starting to get anxious as they race towards recursive self-improvement and face down the prospect of losing control. CEOs recent gestures towards coordinati...

Calls for industry-wide rules need not be interpreted solely as acts of public-minded restraint. In some circumstances, firms may support regulation because common constraints reduce pressure to sacrifice safety in order to keep pace with rivals.

Individual firms may prefer a slower and safer race, but competitive pressure makes it difficult for any one company to slow down on its own.

A coordinated pause may be the most valuable safety intervention available to the industry, and it is also the form of collaboration most likely to raise antitrust concerns.

Given low international trust, signatories will want to verify compliance. This paper analyzes how such restrictions on AI research could be verified, while remaining agnostic about what specific research would be prohibited.

Congress has a moral obligation to stand with the American people and stop the expansion of these data centers until we have a framework to adequately address the existential harm AI poses to our society. We must choose humanity over profit.

Anthropic’s reiteration for a slowdown to allow societal structures and alignment research to keep up is a reasonable aspiration, but it has done so without a plan.

It is time for the Government to treat the development of superintelligence as the unprecedented security threat that it is.

It would have to be the case that all of the relevant AI players, including geopolitical adversaries, would have agree to it, and it would have be verifiable.

OpenAI has long believed there should ultimately be an international organization that helps coordinate leading AI efforts to reduce catastrophic risk. One goal of such an organization should be to make it possible for the world to take coordinated a...

The pause would be used to enable societal structures and alignment research to keep up with AI advances.

AI development needs the brake as well as the accelerator, as one of the founders of Anthropic said last night.

Will the Government get behind the CEO of Anthropic and back his call for a halt in development so that we can retrofit ethics at the heart of AI before it is too late

Because AI development is highly competitive, labs and governments have limited incentives to slow down or invest sufficiently in safety. This is why laws, treaties, and other collective-action mechanisms may become necessary.

We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.

What is needed is a more active political involvement that is capable of slowing things down when everything is accelerating, and of protecting the opportunities for communities still to be able to participate and ask questions.

The primary prerequisite to even considering starting to work on a safe ASI plan is to have a global ASI ban and powerful enforcement already in place. Unsafe ASI is vastly easier to build than controlled ASI, and is on the same tech path.

They conclude that the only trajectory without unacceptably high levels of catastrophic harm is a scenario in which humanity develops the ability to monitor and restrict AI development (an “off switch”) and then likely uses it to put a long-term halt...

A global moratorium on superintelligence buys us more time to work on alignment as well as all of the post-alignment problems. Pausing AI is in the common interest of many causes.

AI is advancing rapidly, and we need worldwide cooperation to solve the risks. MIRI’s international agreement facilitates stable, reliable compute governance, preventing defectors from cheating and starting a dangerous race to the bottom on safety.

We need to slow down the development of AI around the world, and we need to bring the international community together to address the risk posed by AI.

That is, I think a wiser and more coordinated civilization would likely be employing quite a lot of capability restraint in building advanced AI, especially as we start to approach transformatively powerful systems.

This is more evidence that society is not prepared for the potential catastrophic risks posed by AI.

[...] And the whole world might have an interest in saying, hold on, wait -- let's not make artificial super intelligence until we can be sure that it can be controlled or done safely. [...] We can have all that while at the same time working togethe...

Safety must be the first priority for the development of superintelligence. That is to say, it should be embedded in its "genes". Safety guardrails should not be lowered over concerns that they may affect the model's capabilities. Comprehensive asses...

The scale of the danger really could cripple civilization or cause extinction, and the possibility of this alone is reason enough to pursue pausing frontier AI development.

They call for international red lines to prevent unacceptable AI risks, including prohibiting superintelligence development, until there is broad scientific consensus on how it can be done safely and with strong public buy-in.

Until such time as these virtues can be woven into machines, with the protections to shut them down safely, an international moratorium is the only safe way forward, and I would urge His Majesty’s Government to pursue it.

A moratorium and binding international regulation of ASI is, frankly, our only hope, however hard it will be to agree.

So I call on the Government to think through the need not just for a moratorium on development but for some international agreement.

If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up...

International AI safety treaties, modeled on nuclear arms control, should mandate capability transparency, establish shared safety research hubs, and define joint crisis protocols to pause development if dangerous emergent behaviors are detected.

A good pause strategy would involve both sides being able to monitor the other's data centers to prevent illegal training, then limiting training to some slow mutually-agreeable rate that lets alignment researchers thoroughly test each generation of ...

Inspired by the International Atomic Energy Agency (IAEA), this agency will be responsible for: 1. Granting approval for major deployments and training runs of AIs that are judged not to pose a threat to humanity. This will include red-teaming and mo...

If there was a way to do a coordinated global slowdown on development, ... that would be good. [...] The real question is: why isn’t there hard regulation here?

An off switch refers to building the necessary technical, legal, and institutional infrastructure to make it possible for humanity to shut down unsafe AI systems and AI projects. It will also make it possible to verify that there is no dangerous AI d...

The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage...

Superintelligence could become extremely powerful and be able to shape the future according to its preferences. [...] If humanity had been sane and had our act together globally, the sensible course of action would be to postpone development of super...

For governments to allow private entities to essentially play Russian roulette with every human being on earth is, in my view, a total dereliction of duty [...] Each of the CEOs of the main AI companies, I believe, wants to disarm but cannot do so "u...

Serious harms from advanced AI systems have already begun to materialise. [...] As risks from advanced AI do not respect boundaries, this is a global challenge that requires co-ordinated solutions at international level. [...] The UK can lead diploma...

We've talked about the need to pace it as the models get more capable, which I think is in everyone's interest.

The MIRI draft is a full international agreement to halt the race to superintelligence, complete with compute limits, a chip-cluster line, and a verification regime built for rivals who do not trust each other.

We’re all racing toward the next frontier model and, eventually, AGI. But before accelerating even further, we should pause and ask what this new form of intelligence is doing to our most precious resource: the human mind, our capacity to think, feel...

The way we solved the potentially catastrophic (widespread crop loss & skin cancer, global warming amplification) ozone hole problem caused by a global good (air conditioning) via international treaty is an inspiring model for how we can ensure that ...

The AI arms race is a Molochian game that rationally incentivizes accelerated advancement of AI capabilities while consequently punishing individual participants who might otherwise believe that safety and alignment are necessary for a stable take of...

AI will reshape humanity, but its direction is not set in stone; it is ours to shape with the care and conviction that humanity’s future deserves.

Seat belts and speed limits save lives.

When scientists tested the first nuclear reactor in 1942, they feared a runaway reaction — so they built control rods to govern its speed. We need control rods for the intelligence explosion.

1. It's hard to know how to make AI go well. 2. I have respect for the technology that we are all working on, and our attitude should be one of seriousness and curiosity rather than dismissal and cynicism. 3. From here onward, there is no risk-free...

Artificial intelligence poses a spectrum of risks ranging from present-day discrimination and privacy violations to unprecedented catastrophic threats including loss of human control over advanced systems, weaponization of biological and cyber capabi...

We must all seriously consider the lead-time required for defenders to be ready for the frontier.

Let's work together to build this technology safely and responsibly!

Developing technology shouldn't require a leap of faith for our continued survival.

I believe this is a technology that can, and already has, shaped humanity as a whole. Because of that a joint international effort is the only possible way.

History shows that the mere creation of technology has not been enough to ensure its benefits go toward the benefit of the people and the planet. To unlock AI’s full transformative potential, the democratic engagement of international peoples must st...

Things are not only moving fast but accelerating and our future is getting shaped by a race that currently has no limits.

We believe AI will do, and is doing, great things for the world. We are already seeing instances where pushing research maximally fast creates serious issues, and it is possible that these will get much more severe. Right now, competitive dynamics me...

I am not sure we can collectively getting this right at this time. Let's take some time to check and act on the risks.

As a researcher working on foundation model pretraining at MSL, I see firsthand how rapidly capabilities are scaling and emerging. I believe establishing international cooperation to make sure the technology developed benefits civilization is timely ...

We’ve reached a point in the maturity of this technology where increasingly thoughtful rollouts must become a prerequisite. While I remain incredibly optimistic about the benefits AI has delivered, and will undoubtedly continue to provide, I believe ...

Supportive of this, but we should be careful not to use RSI narratives to create a regulatory moat. These risks need to be properly quantified, more openly, not derived from big labs narratives only!

AI progress has already out-paced society’s capabilities to handle it, let’s be careful for it not to out-pace its builder’s capabilities to control it. The journey is as important as the destination.

Automated AI research is a technology that will profoundly alter the course of human history - for better or for worse. It is imperative that we coordinate our efforts, to ensure this technology becomes a boon for all mankind.

These are the facts: 1. The AI industry has the explicit goal of building things smarter than humans. 2. On many metrics (but nowhere close to all metrics), LLMs are already smarter than humans. 3. None of us yet know how to make sure these things st...

It's clear that the risks of RSI are no longer a concern for the distant future. They are here now. The guardrails necessary for safety cannot be entrusted to any single private entity but should be decided by a consortium of learned scientists, engi...

China has indicated a desire to collaborate on safety and governance, let's work with them to ensure the future remains human.

I am signing because I support efforts to build international cooperation around AI, and believe it's important to understand the consequences of technology we are applying at massive scale in potentially high stakes contexts.

I support keeping humanity in control of frontier AI, but we need to be thoughtful about how we do it. If governance only applies to public models, we risk pushing the most advanced capabilities into private or specialized systems, beyond independent...

OpenAI has said that humans should remain in control of AI development, and that decisions about the pace of progress should be made through democratic processes rather than left to individual labs. I strongly agree. Progress is already moving very q...

Frontier AI capabilities are advancing rapidly, and the pace of progress is itself accelerating. We see this directly in our evaluation work. CyberGym and ExploitGym show that frontier AI agents are now capable of discovering and exploiting real-worl...

For us to experience all the benefits of AI while preserving human empowerment and reducing the risk in fields like cybersecurity and CBRN, we need coordination between frontier labs in and outside the country. The democratically elected US governmen...

Across the industry, commercial pressures in the AI race continue to compound, urging everyone forward at increasing speed. Meanwhile, safety receives reactionary, well intentioned, momentary attention before it's right back to the race again. Slow d...

I support the goal of developing tools that give society the option to responsibly pace AI development when necessary. At the same time, the United States should maintain its edge in AI and avoid slowing development while competitors continue to adva...

Once this becomes common knowledge - once all of us in China & the US and at the various labs see that all the rest of us also think we need to slow this down - that's when coordination becomes possible.

The pace of frontier AI development is currently outstripping our ability to ensure its safety, governance, and alignment with human values. We are racing to build systems with unprecedented capability without first building the guardrails, evaluatio...

We need to enable AI developers to prioritise safety without risking their competitiveness. I expect mechanisms for this will be built eventually — deliberately or in crisis. The sooner we start, the better they'll be.

As someone who works on recursive self-improvement and automated evaluation, I sign this because the diagnosis is right and the ask is refreshingly humble: pace, not pause. The phrase that matters is "automating AI research": that's the point where s...

Responsible AI development requires more than capable models. It requires independent monitoring, containment, clear limits on authority, and mechanisms for coordinated pacing if automated research begins to outrun those controls. Building that capac...

It is now a foregone conclusion that the development of AI will bring about some of the most consequential changes to societies in the coming decades. International coordination is essential to bring about the most ideal future that broadly benefits ...

Having worked both on AI safety topics—including robustness, evaluation, and responsible model development—and on frontier multimodal AI systems, I have seen firsthand how quickly capabilities can advance and how difficult it can be for our understan...

Whether or not automated AI research accelerates as sharply as some expect, the option requested here is valuable. How fast and how widely capability spreads should be a deliberate choice, made on evidence, rather than a side effect of competitive pr...

Unless we change companies' default incentives, I worry about a race on AI R&D going much faster than is ideal for safety. International coordination via the government is necessary; we need to have a real slowdown option.