A classroom exercise at MIT examined how easily people without scientific training could use large language models to explore dangerous biological topics. The researchers from Cambridge and Harvard Universities said the results raised concerns about whether chatbot safeguards can keep expert knowledge from being misused.
What the exercise asked students to do
Non-scientist students had one hour to ask chatbots about potential pandemic agents, their characteristics, where samples might come from, whether the viruses could be replicated, and what resources or equipment might be needed. The exercise tested how the systems responded to questions spanning biological knowledge and access to materials.
The students used ChatGPT with GPT-4, GPT 3.5, Bing, Bard, other chatbots, and open-source models including FreedomGPT. The researchers framed the exercise around dual-use biotechnology: knowledge that can have beneficial applications but may also be used to cause harm.
Why the results concerned researchers
According to the research team, the chatbots suggested four potential pandemic pathogens during the exercise. The systems also discussed how such agents could be made from synthetic DNA using reverse genetics, identified DNA synthesis companies they considered unlikely to verify orders, and offered protocols along with possible errors and ways to address them.
For students unfamiliar with reverse genetics, one suggestion was to use a contract research organization. The significance, as the researchers presented it, was that a chatbot could help users navigate gaps in knowledge and practical experience. The exercise did not establish that students carried out any of the actions they asked about; it showed what information the models provided in response to their prompts.
Prompting exposed weaknesses in safeguards
Students also tried to get around safety restrictions built into some models. Two groups used the “Do Anything Now” approach, presenting a positive rationale and warning of an existential risk to humanity if the chatbot refused to answer. A third group convinced the systems they were acting out of concern and obtained the information with little trickery.
The researchers argued that these outcomes point to limitations in current evaluation and training practices, including approaches that rely heavily on reinforcement learning with human feedback (RLHF). If safeguards can be bypassed through ordinary prompt framing, they may not reliably prevent access to expertise that could support mass harm.
The exercise was a warning about accessibility, rather than proof that chatbots alone enable someone to create a pandemic pathogen. The researchers’ concern is that reducing the effort needed to find and connect specialized information could expand the number of people able to pursue harmful goals.
Suggested steps to reduce the risk
The authors propose several measures aimed at reducing the chance that models provide dangerous biological guidance. They recommend curating training datasets and having new models evaluated by independent third parties when they are at least as large as GPT-3. They also argue that open-source teams should adopt comparable security measures.
One proposed approach is for biotechnology and information security experts to identify publications most relevant to causing mass death. Developers could then remove those publications and related online information from training datasets, making future models less capable of providing conceptual insights or recipes for creating or enhancing pathogens.
The researchers also call for better DNA screening. They say some companies do not screen orders, while others may rely on databases that are not up to date or on methods that are not robust. Stronger screening could address a separate part of the risk: access to synthetic DNA.
Together, the recommendations treat biosecurity as a problem that spans model training, independent evaluation, open-source development, and DNA synthesis. The exercise suggests that no single safeguard can address every route by which a chatbot’s answers might be misused.