Ban open source AI models capable of creating WMDs

If mandated AI safety reviews are to have any teeth, they must be accompanied by a ban on dangerous AI models that have failed their assessments. This means that some method must be found to restrict access to dangerous open weight AI models. [...] C...

Open models can be easily modified for harm. [...] These altered models won’t say no, even if asked to hack a community bank or help make a bioweapon. [...] The U.S. should seek an agreement with China that neither country will release model weights ...

Capability testing must be the primary input for open-release decisions because open-weight mitigation remains unreliable, and labs and governments must identify models that are too dangerous for unrestricted release.

However, we recommend against publicly releasing frontier model weights.

Open-source AI-bio models possess potentially global, indiscriminate harm to states, people, animals, and the environment if not properly anticipated. While individual state actors may pursue their own BW programs, it is in no state’s interest to all...

Advanced model weights can be high-consequence assets. If stolen or openly released without adequate safeguards, they may allow actors to remove safety filters, fine-tune models for harmful uses, or reproduce capabilities that would otherwise remain ...

Implement conditional bans on Chinese open-weight models. [...] The more adversarial recommendations should be reserved for labs that release frontier open-weight models without adequate safeguards, and conditional on dual-use evaluations by the U.S....

Today, an even greater risk than a naturally caused pandemic is that a non-government group will use open source AI tools to design a bioterrorism weapon.

Someday AIs really will be able to make nukes or pull off $500 million hacks. At that point, companies will have to certify that their model has been trained not to do this, and that it will stay trained. But if it were open-source, then anyone could...

But open sourcing, that's just sheer catastrophe. [...] wrong approach, the wrong ideal. [...] where if you could align it it would take time [...] not kill everyone.

When you open source a big model and give people the weights [...] You get people using that open source model and training it, refining it to do something bad like cyberattacks or making biological weapons because it's much easier to refine the mode...

I think it's really important, because when we put open source out there for something that could be dangerous, which is a tiny minority of all the code that's open source, essentially we're opening the door to all the bad actors. And as these system...

The term “open-weight AI model” means an AI model for which the model weights are made publicly available without restriction on access. [...] “meaningful uplift” means specific operational assistance that would materially advance an adversary’s CBRN...

There are some digital products (say CSAM and AI letting terrorists make bioweapons) that I oppose regardless of whether they are open-source or not, but I'm overall supportive of open source, and you can easily verify that my MIT research group defa...

endowing rogue nations or terrorists with tools to synthesize a deadly virus. [...] keep the “weights” of the most powerful models out of the public’s hands.

On the current course, the leading Chinese AGI labs won’t be in Beijing or Shanghai—they’ll be in San Francisco and London. In a few years, it will be clear that the AGI secrets are the United States’ most important national defense secrets—deserving...

ASL-2 refers to systems that show early signs of dangerous capabilities – [...] give instructions on how to build bioweapons – [...] e.g. a search engine couldn’t.

On the issue of open source, you each raised the security and safety risk of AI models that are open source or are leaked to the public, the danger. There are some advantages to having open source, as well. It's a complicated issue. I appreciate that...

Finally, to avoid that open-source models are accessed and retrofitted for malicious purposes a potential solution is to create self-destruct codes that activate if the model is tampered with.

They tried to pass laws to stifle [free open-source AI software]. ... After that narrative lost credibility, and our government no longer believes that, thank God, the same people said, “Oh, AI could create bioweapons.” After the RAND and other studi...

A safety regime that restricts legitimate defenders, while capable models remain available for attackers, creates an asymmetric disadvantage.

To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category. We should instead focus on keeping powerful chips out of authoritarian hands, stopping industrial-scale distillation, and req...

However, the solution here is absolutely not to restrict open model development in the name of security—cyber, national, or otherwise.

Since computational infrastructure is largely open-access, decentralized, and global, regulatory chokepoints are limited. Export controls may delay access to high-performance computing, but they are unlikely to prevent the use of open-source models f...

current evidence is not sufficient [...] (but) take steps to ensure that the government is prepared to act if heightened risks emerge.

there has to be full open source here. [...] if you're worried about AI-generated pathogens, [...] Let's do a Manhattan Project for biological defense.

But today, going, banning open source, preventing it from happening is really a way, well, to enforce regulatory capture, even though the actors that would benefit from it don’t want it to happen. […] If you actually ban small actors from doing thing...

current evidence is not sufficient [...] (but) take steps to ensure that the government is prepared to act if heightened risks emerge.

That is not a reason to withhold it. It is a reason to expect the same failure and to build for it in advance.

CDT appreciates that the NTIA correctly concluded that there is not yet enough evidence of novel risks from open foundation models to warrant new restrictions on their distribution.