Advances in artificial intelligence-assisted protein engineering are enabling breakthroughs in protein design, but they also introduce biosecurity challenges related to potential production of harmful proteins. Though screening software to detect harmful proteins exists, a new multi-month analysis of such software reports that this software has vulnerabilities; some proteins of concern could evade detection. Critically, the study also offers a way to improve detection rates of proteins of concern going forward. AI-assisted protein design (AIPD) enables powerful advances in medicine and biology, enabling researchers to modify existing proteins or design wholly new ones with novel structures and functions. However, this powerful technology could also be misused to design harmful proteins. A necessary step to make a protein in the lab is ordering the DNA that encodes it. The companies that provide these synthetic nucleic acids screen customer orders with biosecurity screening software (BSS) to identify and block genes that encode proteins of concern. However, protein sequence generative models can produce functional variants in which amino‑acid sequences differ enough from controlled examples to evade detection. Despite this, no systematic assessment of BSS vulnerabilities has been undertaken, and international governance regarding the potential biosecurity risks of generative protein design is lacking. These concerns were previously highlighted in Science in a Policy Forum by Bloomfield et al. [https://www.science.org/doi/full/10.1126/science.adq1977], as well as in an Editorial by David Baker and George Church [https://www.science.org/doi/10.1126/science.ado1671].

In this study, Bruce Wittmann and colleagues employ an “AI red teaming” approach to evaluate BSS models with the goal of improving them to enhance biosecurity. Using open-source AI protein design software, Whittmann et al. generated more than 75,000 variants of hazardous proteins and submitted them to four different BSS developers and found that, while all tools performed nearly flawlessly when screening the original wild-type proteins, their ability to detect reformulated variants was inconsistent. According to the authors, the results suggest that, although current BSS systems remain effective for unaltered sequences, they lack consistent sensitivity when faced with protein sequence homologs engineered using modern generative AI methods, despite being similar. Following the initial findings, and in collaboration with BSS providers, Wittmann et al. developed software patches, which were deployed by three of the four BSS providers’ systems. These updates resulted in improved detection rates for AI-generated variants without significantly increasing false positives. Nonetheless, the authors note that no tool achieved complete coverage: across providers, about 3% of the variants most likely to retain functionality still escaped detection. ““AI advances are fueling breakthroughs in biology and medicine, yet with new power comes the responsibility for vigilance and thoughtful risk management,” said Eric Horvitz, senior author of the study and Microsoft’s chief scientific officer. “Beyond identifying and working to mitigate this specific vulnerability, our aim was to develop and demonstrate an effective process: building a cross-sector team, applying rigorous scientific methods, and creating a framework for sharing sensitive data and insights in ways that advance the science while managing potential risks.””

The authors of this work have made Science aware that parts of the data and code should not be made available in a public repository due to potential for misuse. The authors have thus designed a tiered access scheme for data release in which interested parties can request access to the restricted material by contacting designated representatives from the International Biosecurity and Biosafety Initiative for Science (IBBIS), a non-profit organization (https://ibbis.bio). The requestor will need to provide their identity, affiliation, and a brief explanation for their use of the data and will be subject to a data usage agreement. A committee at IBBIS will evaluate applications and determine if the requestor information provided is accurate and the proposed usage is legitimate. The restricted data is organized in tiers according to information hazard level, with the highest tier including code used in the study, and it will be released according to suitability for the proposed usage. The Science editors have agreed the proposed scheme adequately balances the security concerns with the data and code availability requirements in Science Editorial Policies. The authors have also made provisions for future declassification and/or succession of data management. Any concerns about the availability of data through this scheme can be raised by contacting science_data@aaas.org.

***A related embargoed news briefing was held on Tuesday, 30 September, as a Zoom webinar. A recording, with transcript, can be found at https://aaas.zoom.us/rec/share/75vhLzuQGdN4nE16fEhGGqNNzHIXAEBIbvm1OwOsW-9Wjkr57kOhkmFLicUF-h4E.PelPeKq8ffXCVeg0 - the passcode is ^q707eRd. See Other Information for Journalists section for full briefing information.***