Ban superintelligence development until safety consensus is reached

If... we were to find that we were reaching a critical capability level and we didn't have the appropriate mitigations, then we would not be launching.

We recommend an international deal between all major world powers to avoid a dangerous race to superintelligence.

If you delay the advent of superintelligence, then that gives society more time to prepare and more time to solve the various problems that it represents.

That leave be given to bring in a Bill to make provision to prohibit the development, deployment and operation of artificial superintelligence systems; to establish monitoring and control powers in respect of such systems; and for connected purposes.

The Nakada Foundation is building the political case for a binding international agreement to halt the race to superintelligence before it is built.

The premature development of artificial superintelligence poses major risks to humanity, so researchers have proposed international agreements halting such development until it can be done safely.

The UK therefore should begin by prohibiting the development of superintelligence on British soil and then work to build a coalition of states agreeing to prohibit it globally.

We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.

If anyone, anywhere builds a superhuman artificial intelligence using present methods, the most likely outcome is catastrophe.

Article I prohibits signatories from developing or deploying artificial superintelligence and from permitting others within their jurisdiction to work towards that end.

Article I prohibits signatories from developing or deploying artificial superintelligence and from permitting others within their jurisdiction to work towards that end.

We believe that in order to do this, we must secure an international prohibition on its development.

They conclude that the only trajectory without unacceptably high levels of catastrophic harm is a scenario in which humanity develops the ability to monitor and restrict AI development (an “off switch”) and then likely uses it to put a long-term halt...

Superintelligence should be banned.

A global moratorium on superintelligence buys us more time to work on alignment as well as all of the post-alignment problems. Pausing AI is in the common interest of many causes.

Superintelligence would likely break the very operating system of human civilization – and is completely unnecessary. If we instead focus on building controllable AI tools to help real people today, we can far more reliably and safely realize AI’s in...

If we race to build superintelligence without clear and morally informed parameters, we risk undermining the incredible potential AI has to alleviate suffering and enable flourishing. We should intentionally harness this amazing technology to help pe...

This is not a ban or even a moratorium in the usual sense. It’s simply a proposal to require adequate safety measures for a technology that, according to its developers, has a significant chance to cause human extinction. Is that too much to ask?

To get the most from what AI has to offer mankind, there is simply no need to reach for the unknowable and highly risky goal of superintelligence, which is by far a frontier too far. By definition this would result in a power that we could neither un...

Frontier AI systems could surpass most individuals across most cognitive tasks within just a few years. These advances could unlock solutions to major global challenges, but they also carry significant risks. To safely advance toward superintelligenc...

Yeah, we want specific AI tools that can help cure diseases, strengthen national security, etc. But does AI also need to imitate humans, groom our kids, turn us all into slop junkies and make zillions of dollars serving ads? Most people don’t want th...

95% of Americans don’t want a race to superintelligence, and experts want to ban it.

The future of AI should serve humanity, not replace it. The true test of progress will be not how fast we move, but how wisely we steer.

AI offers extraordinary promise to advance human rights, tackle inequality, and protect our planet, but the pursuit of superintelligence threatens to undermine the very foundations of our common humanity. We must act with both ambition and responsibi...

Congress has a moral obligation to stand with the American people and stop the expansion of these data centers until we have a framework to adequately address the existential harm AI poses to our society. We must choose humanity over profit.

That branch of AI is lethal. We've got to do something about that. Nobody should build it. And everyone needs to agree to that by treaty. Treaties are not perfect, but they are the best we have as humans.

We need to slow down the development of AI to give democracy, to give ordinary people, a chance to catch up.

We call on the Government to seek a global agreement that bans the development of artificial superintelligence and to encourage the establishment of a globally accepted safe path for developing advanced AI, ensuring strong safety standards and intern...

Development of superintelligence should be prohibited until there is broad scientific consensus that it can be done safely and controllably, and there is strong public buy-in.

More and more parliamentarians seem ready to listen to the argument that what has previously been a headlong race to build superintelligence should be subject to a global, verifiable moratorium.

[...] And the whole world might have an interest in saying, hold on, wait -- let's not make artificial super intelligence until we can be sure that it can be controlled or done safely. [...] We can have all that while at the same time working togethe...

It is better to slow down a bit to lay a solid foundation for safety, than to seek quick success and instant benefits that might lead human society into an irreversible and perilous situation.

If AIs became strategically competent enough, they may realize that RSI is too dangerous because they're not good enough at alignment or philosophy or strategy, and potentially convince, help, or work with humans to implement an AI pause.

We need to course correct immediately and work to prevent the development of superintelligent AI, and we need to do this internationally, which will take time. We cannot afford to wait for more evidence that the risk is imminent.

The UK can lead diplomatically in recognising a moratorium, with verifiable commitments from all major AI-developing nations.

I urge the Minister to formally acknowledge extinction risk from superintelligent AI as a national security priority and to lead on international efforts to prohibit superintelligence development.

So I call on the Government to think through the need not just for a moratorium on development but for some international agreement.

They call for international red lines to prevent unacceptable AI risks, including prohibiting superintelligence development, until there is broad scientific consensus on how it can be done safely and with strong public buy-in.

Canada should form a coalition with other countries, including middle powers, and lay the diplomatic groundwork for an international prohibition on the development of superintelligent AI.

That is why we should stop the development of superintelligence before it is too late.

We need to make this a goal state. If the countries come together and have an international summit on how to safely ban superintelligence, but we fail at the diplomacy, that would be unfortunate, but at least we would have tried.

Governments are miles behind; in my view, our Government need to step up. They can start by acknowledging the existential risk of advanced AI and joining the numerous UK parliamentarians - including many in this Room today - who have called for a pro...

There ought to be a law against further escalation of AGI capabilities, trying to halt it short of the point where it births superintelligence. A line drawn sharply and conservatively, because we don't know how much further we can dance across this m...

We don't have to do this. We have human-competitive AI, and there's no need to build AI with which we can't compete. We can build amazing AI tools without building a successor species. The notion that AGI and superintelligence are inevitable is a cho...

A moratorium and binding international regulation of ASI is, frankly, our only hope, however hard it will be to agree. It will be even harder to enforce, but we have to do it; there is no choice.

In the face of these threats, I urge the Government to take the following steps: first, to publicly recognise that superintelligence poses an extinction threat to humanity; secondly, for the UK to prevent the development of superintelligence on its s...

We need to Pause frontier AI development if we're ever going to have time and latitude to even see if superintelligence could be safe and respect human consent. And if we find out it can't be, then we need to keep superintelligence paused forever.

We should rapidly develop powerful AI tools that help cure diseases and solve practical problems, but not autonomous smarter-than-human machines that nobody knows how to control. Creating superintelligent machines is not only unacceptably dangerous a...

If we build it using anything remotely like modern methods, on anything remotely like the current understanding or lack of understanding that we have about AI, then yeah, building it anytime soon would be a death sentence. [...] When you're in a car ...

Until some company or scientist says ‘Here’s the proof! We can definitely have a safety mechanism that can scale to any level of intelligence,’ I don’t think we should be developing those general superintelligences. We can get most of the benefits w...

Where are the laws binding development on superintelligent AI?

If we can, maybe it would be good to have a slightly slower pace than we’re currently predicting, [...] so that we can get this right societally. [...] I think so. [...] international collaboration.

There is AI that is not responding to instructions, to commands, and there are documented cases of that. [...] If we don't consider ways to manage what we know is occurring already, and there is use of autonomous weapons already, if we are not lookin...

Most of the AI safety community has been far too coy about extinction risk. We're not. It's not that complicated: AI smarter than humanity poses an unacceptable risk of human extinction, and we need an [international ban].

The question that the noble Lord posed was: do we therefore go for a moratorium? That would be highly desirable, but I do not think that it will be possible in the short term.

One rather draconian solution to all these problems, proposed earlier this week by Andrea Miotti, would be to create an “International Treaty to Ban Superintelligence.” Why worry about what to do with the latest and greatest models if you can make th...

One example—potentially the most meaningful threshold articulated thus far—would be AI systems that outperform humans in making new breakthroughs in AI and developing increasingly more powerful AI systems. AI systems with such research and developmen...

We’ve already heard calls, including in Parliament, to ‘Pause AI’. I think doing so would be a double betrayal. A betrayal of British talent and British interests.

We are on our way to personal superintelligence: an assistant that can help anyone, anywhere with the things that matter most to them.

Pausing frontier AI development will not solve the problems its supporters claim it will solve. If anything, it risks making several of them worse.

We felt that it wouldn't actually help anyone for us to stop training AI models.

The conclusion is sobering: a global stop to the development of superintelligent AI seems impossible.

My own view, after talking with colleagues in the AI Security Institute and the Alan Turing Institute, is that a moratorium would be unenforceable.

For me, a one-sided moratorium which ties our hands behind our backs while the likes of North Korea and the al-Qaedas of the world crack on: no, I am afraid that that worries me even more.

I hope that, when the Minister responds, she will confirm that the Government have no plan to suppress the development of ASI. Inquiry and discovery are deeply ingrained in the human psyche and the AI revolution we are living through should certainly...

My own view, after talking with colleagues in the AI Security Institute and the Alan Turing Institute, is that a moratorium would be unenforceable. Instead, I support the proposal made this week by the noble Baroness, Lady Harding, to set up a commis...

I come from the position of saying that moratoriums will not work. But we can work in co-operation with other nations that have already started regulating, such as South Korea and Australia, as well as work with our AI Security Institute in the Unite...

I come from the position of saying that moratoriums will not work. But we can work in co-operation with other nations that have already started regulating, such as South Korea and Australia, as well as work with our AI Security Institute in the Unite...

Developing superintelligence is not like playing Russian roulette; it is more like undergoing risky surgery for a condition that will otherwise prove fatal. [...] Models incorporating safety progress, temporal discounting, quality-of-life differentia...

[...] the idea of stopping or even substantially slowing [...] is fundamentally untenable. If one company does not build it [...] authoritarian countries would simply keep going.

Since the launch of powerful open models like the Llama, Falcon, Mistral, and Stable Diffusion families, critics have pressed to keep other such genies in the bottle. "Open source software and open data can be an extraordinary resource for furthering...

We believe in accelerationism [...] upward spiral continues forever. [...] Our enemy is the Precautionary Principle [...] and we must jettison it with extreme prejudice.

My opinion is that the moratorium that we should focus on is actually deployment until we have good safety cases. I don't know that we need to pause that particular project, but I do think it's emphasis on focusing more on AI safety, on trustworthy, ...

This is a formula for outright stagnation. [...] But rather than trying to anticipate every risk, regulators should let entrepreneurship flourish while efforts to monitor and improve AI safety proceed in parallel.

My first reaction to [the letter] is that calling for a delay in research and development smacks me of a new wave of obscurantism,

But the governance of the most powerful systems, as well as decisions regarding their deployment, must have strong public oversight. We believe people around the world should democratically decide on the bounds and defaults for AI systems. Second, w...

My own view, after talking with colleagues in the AI Security Institute and the Alan Turing Institute, is that a moratorium would be unenforceable. Instead, I support the proposal made this week by the noble Baroness, Lady Harding, to set up a commis...

If I could put a pause on AI or really advanced AI, superintelligence, I would. It doesn’t seem that is realistic.

Treating speculative “superintelligence” as the policy target and proposing to freeze development until there’s public buy‑in and scientific consensus distracts from the actual, present‑day harms of AI systems. These systems are already amplifying di...

AI is a foundational technology that every country, every industry, and every person will ultimately rely on. You can’t slam on the brakes and wait for some abstract consensus before building the future. We should not be putting up stop signs to prog...

We are likely to eventually need something like an IAEA for superintelligence efforts; any effort above a certain capability (or resources like compute) threshold will need to be subject to an international authority that can inspect systems, require...