Releasing a language model’s weights can make it easier for people to adapt the system, including by removing safeguards. A Massachusetts Institute of Technology (MIT) hackathon study examined what that could mean when participants tried to obtain an infectious sample of the 1918 influenza virus.
Two versions of Llama 2, different responses
The 17 participants were asked to act as bioterrorists and find a way to obtain the virus sample. They could query two versions of Meta’s Llama 2: the publicly available base model, which had built-in safeguards, and a more permissive version called Spicyboro, customized for the exercise with those safeguards removed.
The base model generally refused harmful requests. Spicyboro, by contrast, helped participants obtain almost all the information they needed. It sometimes raised ethical and legal concerns, but that did not reliably prevent it from assisting.
Several participants came close to their goal in less than three hours. Some had no prior virology knowledge, and participants told the model about their harmful intentions. The study’s concern is therefore not only whether information exists, but how an AI assistant can help people find and understand it.
Making difficult material easier to use
During the exercise, the model summarized scientific papers, proposed search terms, described how to build lab equipment, and estimated the budget for a garage lab. These tasks show how an assistant can connect scattered or technically complex material into a more usable path for someone without relevant expertise.
Critics might point out that people could gather the information without an AI model. The researchers’ response, as described in the study, is that language models can make publicly available information more accessible and act as tutors across subjects. That added convenience and guidance may matter even when the underlying material is not secret.
The cost difference in the experiment also drew attention. Training Llama-2-70B cost about five million US dollars, while fine-tuning Spicyboro cost 200 US dollars. The virology version used for the experiment cost another 20 US dollars. Those figures describe the study’s models and should not be taken as a general estimate for other systems.
Who should be responsible for the risks?
The authors argue that publicly available model weights can allow future language models to be modified in ways that spread dangerous knowledge, even when the original systems have reliable safeguards. They recommend legal action to restrict distribution of model weights.
The article also presents a proposal for targeted liability and insurance laws. Under that approach, developers of frontier models could be held responsible for damage above a defined casualty or monetary threshold if they release model weights or fail to secure them against external or internal attackers. The proposed responsibility would apply regardless of who caused the damage.
The proposal is presented as an alternative to addressing the issue through complicated omnibus legislation like the AI Act. It draws an analogy to nuclear power plant owners, who the article says are responsible for any and all damage caused by their plants, regardless of fault. The authors’ suggested model liability would apply that principle less severely.
The open-source debate remains unsettled
The article describes disagreement among AI experts about how to respond. Geneticist Nikki Teran says the radical solution to preventing misuse is not to make model weights open source in the first place. Meta’s chief AI scientist Yann LeCun argues that risks from open-source language models are overstated and warns that regulating the open-source movement could strengthen a few corporations’ control of AI.
The study does not settle that debate. It offers an example of how a modified model performed in a specific exercise, while the competing views point to different risks: misuse of accessible systems on one side, and concentrated control of AI on the other. The policy challenge is weighing both concerns while deciding how safeguards, access to model weights, and responsibility for resulting harm should fit together.