Ban open source AI models capable of creating WMDs

Someday AIs really will be able to make nukes or pull off $500 million hacks. At that point, companies will have to certify that their model has been trained not to do this, and that it will stay trained. But if it were open-source, then anyone could...

endowing rogue nations or terrorists with tools to synthesize a deadly virus. [...] keep the “weights” of the most powerful models out of the public’s hands.

There is no permanent way to apply safety limitations to prevent users from obtaining help from the model with regard to bioweapons-related tasks.

And a second point is about liability. And it's not completely clear where exactly the liability should lie. But to continue the nuclear analogy, if a corporation decided they wanted to sell a lot of enriched uranium in supermarkets, and someone deci...

I think it's really important because if we put something out there that is open source and can be dangerous – which is a tiny minority of all the code that is open source – essentially we're opening all the doors to bad actors [...] As these systems...

Let's open source nuclear weapons too to make them safer. The good guys (us) will always have bigger ones than the bad guys (them) so it should all be OK.

I fully expect that in a few years it’s going to be completely obvious to everyone that open-sourcing AI is just not wise.

Any and all safeguards can and will be removed within days of a large model being open-sourced.

[What] was frightening about this demonstration was how readily available some of this content or software is on kind of the black market right now, and how it can be weaponized and used to manipulate people, destroy lives and build weapons of mass d...

There are some digital products (say CSAM and AI letting terrorists make bioweapons) that I oppose regardless of whether they are open-source or not, but I'm overall supportive of open source, and you can easily verify that my MIT research group defa...

Today, an even greater risk than a naturally caused pandemic is that a non-government group will use open source AI tools to design a bioterrorism weapon. [...] We'll need to be deliberate about how this technology is developed, governed, and deploye...

Some of the most recent models maybe can help people make biological weapons.

Our Preparedness Framework defines capability thresholds that could lead to severe risk, and risk-specific guidelines to sufficiently minimize the risk of severe harm. For biology, a model that meets a High capability threshold could provide meaningf...

Via the Australia Group and the US Department of Commerce, the US Government should explicitly design export controls to limit open sourcing of the riskiest AI-enabled Biological Design Tools (BDTs). Since publishing a tool online can be considered ...

SB 1047 would, for example, forbid the release of a frontier model that could be easily induced to output detailed instructions for making a bioweapon.

And @pmarca would you open source the manhattan project? This one is more serious for national security. We are in a tech economic war with China and AI that is a must win. This is exactly what patriotism is about, not slogans.

On the current course, the leading Chinese AGI labs won’t be in Beijing or Shanghai—they’ll be in San Francisco and London. In a few years, it will be clear that the AGI secrets are the United States’ most important national defense secrets—deserving...

You basically have a bomb that you're making available for free, and you don’t have any way to defuse it necessarily. It’s just an obviously fallacious argument. We didn’t do that with nuclear weapons: we didn’t say ‘the way to protect the world fro...

The thought of open source AGI being released before we have worked out how to regulate these very powerful AI systems is really very scary. In the wrong hands technology like this could do a great deal of harm. It is so irresponsible for a company t...

Widely releasing the very advanced AI models of the future would be especially problematic, because preventing their misuse would be essentially impossible, he says, adding that they could enable rogue actors and nation-state adversaries to wage cybe...

ASL-2 systems show early signs of dangerous capabilities – for example ability to give instructions on how to build bioweapons – but where the information is not yet useful due to insufficient reliability or not providing information that e.g. a sear...

But open sourcing, you know, that's just sheer catastrophe. The whole notion of open sourcing, this was always the wrong approach, the wrong ideal. There are places in the world where open source is a noble ideal and building stuff you don't understa...

On the issue of open source, you each raised the security and safety risk of AI models that are open source or are leaked to the public, the danger. There are some advantages to having open source, as well. It's a complicated issue. I appreciate that...

Artificial intelligence is advancing so rapidly that many who have been part of its development are now among the most vocal about the need to regulate it. While AI will bring many benefits, it is also potentially dangerous; it could be used to creat...

Once the model weights are released, other less responsible actors can easily modify the model to strip away its safeguards.

Content controls, a free content filter, monitoring of applications, and a code of conduct are several other steps industry and academia, with the coaxing of the Administration and policymakers, could take to encourage responsible science and guard a...

From a business perspective, the difference between open and closed is a little bit overblown. From a security perspective, the difference between open and closed models is, for some intents and purposes, overblown. The most important thing is how po...

In a time dominated by open-weight biological AI models developed across the globe, limiting access to sensitive pathogen data to legitimate researchers might be one of the most promising avenues for risk reduction.

We'll need to be rigorous about mitigating these risks and careful about what we choose to open source.

LLMs, such as GPT-4 and its successors, might provide dual-use information and thus remove some barriers encountered by historical biological weapons efforts. [...] BDTs may enable the creation of pandemic pathogens substantially worse than anything ...

Finally, to avoid that open-source models are accessed and retrofitted for malicious purposes a potential solution is to create self-destruct codes

To try to ban Americans from using open source, open weight, Chinese models or other open models would be handing a monopoly to certain American companies on a platter.

I see the danger of this concentration of power through proprietary AI systems as a much bigger danger than everything else. What works against this is people who think that for reasons of security, we should keep AI systems under lock and key becaus...

We've always made the assessment that the benefits to the world far outweigh the dangers. [...] as capabilities increase, I think that's going to be an important question to keep considering.

AI is just an information processing and retrieval system akin to a search engine, which can be used in many ways. [...] The fact that criminals use them is a corollary of what AI models are: namely, tools.

Since computational infrastructure is largely open-access, decentralized, and global, regulatory chokepoints are limited. Export controls may delay access to high-performance computing, but they are unlikely to prevent the use of open-source models f...

As the Office of the Director of National Intelligence has assessed, AI models have the potential to enable advanced military and intelligence applications; lower the barriers to entry for nonexperts to develop weapons of mass destruction (WMD); supp...

Open source AI has the potential to unlock unprecedented technological progress. It levels the playing field, giving people access to powerful and often expensive technology for free, which enables competition and innovation that produce tools that b...

current evidence is not sufficient [...] (but) take steps to ensure that the government is prepared to act if heightened risks emerge.

there is not yet enough evidence of novel risks from open foundation models to warrant new restrictions on their distribution.

In their report, NTIA rightly notes that “current evidence is not sufficient to definitively determine either that restrictions on such open-weight models are warranted or that restrictions will never be appropriate in the future.” Instead of recomme...

open source is not more dangerous than closed source

there has to be full open source here. [...] if you're worried about AI-generated pathogens, [...] Let's do a Manhattan Project for biological defense.

But today, going, banning open source, preventing it from happening is really a way, well, to enforce regulatory capture, even though the actors that would benefit from it don’t want it to happen. […] If you actually ban small actors from doing thing...

Encouraging companies to keep the details of their models secret is likely to lead to “serious downstream consequences for transparency, public awareness, and science.” […] Anyone in the world can read them and develop their own models, she says.

My view is that the hype has somewhat run ahead of the technology. I think a lot of the existential warnings relate to models that don’t currently exist, so-called super-intelligent, super-powerful AI models – the vision where AI develops an autonomy...

Regulating the open-source release of GPAI models is broadly unnecessary