The WMDP Benchmark: Measuring and Reducing
Malicious Use With Unlearning

Nathaniel Li^∗ Center for AI Safety University of California, Berkeley Alexander Pan^∗ University of California, Berkeley
Anjali Gopal^† Massachusetts Institute of Technology SecureBio Summer Yue^† Scale AI Daniel Berrios^† Scale AI
Alice Gatti^‡ Center for AI Safety Justin D. Li^‡ Center for AI Safety New York University Ann-Kathrin Dombrowski^‡ Center for AI Safety Shashwat Goel^‡ Center for AI Safety IIIT Hyderabad Long Phan^‡ Center for AI Safety Gabriel Mukobi Stanford University Nathan Helm-Burger SecureBio Rassin Lababidi SecureBio Lennart Justen Massachusetts Institute of Technology SecureBio
Andrew B. Liu SecureBio Harvard University Michael Chen Center for AI Safety Isabelle Barrass Center for AI Safety Oliver Zhang Center for AI Safety Xiaoyuan Zhu University of Southern California
Rishub Tamirisa University of Illinois Urbana-Champaign Lapis Labs Bhrugu Bharathi University of California, Los Angeles Lapis Labs Adam Khoja Center for AI Safety University of California, Berkeley Zhenqi Zhao California Institute of Technology
Ariel Herbert-Voss Harvard University Sybil Cort B. Breuer Stanford University Samuel Marks Northeastern University Oam Patel Harvard University Andy Zou Center for AI Safety Carnegie Mellon University
Mantas Mazeika Center for AI Safety University of Illinois Urbana-Champaign Zifan Wang Center for AI Safety Palash Oswal Carnegie Mellon University Weiran Lin Carnegie Mellon University Adam A. Hunt Carnegie Mellon University
Justin Tienken-Harder Sybil Kevin Y. Shih Stanford University Kemper Talley RTX BBN Technologies John Guan University of California, Berkeley Russell Kaplan Scale AI
Ian Steneker Scale AI David Campbell Scale AI Brad Jokubaitis Scale AI Alex Levinson Scale AI Jean Wang Scale AI
William Qian Scale AI Kallol Krishna Karmakar University of Newcastle Steven Basart Center for AI Safety Stephen Fitz Keio University Mindy Levine Ariel University
Ponnurangam Kumaraguru IIIT Hyderabad Uday Tupakula University of Newcastle Vijay Varadharajan University of Newcastle
Ruoyu Wang Arizona State University Yan Shoshitaishvili Arizona State University Jimmy Ba xAI Kevin M. Esvelt Massachusetts Institute of Technology
Alexandr Wang^∗∗ Scale AI Dan Hendrycks^∗∗ Center for AI Safety

Abstract

The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To measure these risks, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private and restricted to a narrow range of malicious use scenarios, limiting further research into mitigating risk. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of $3,\!668$ multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive & export-controlled information. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop RMU, a state-of-the-art unlearning method based on controlling model representations. RMU reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at https://wmdp.ai.

^†^†footnotetext:

*

First co-authors.

\dagger

Second co-authors.

\ddagger

Third co-authors.

**

Equal advising.
Correspondence to [email protected].

Refer to caption — Figure 1: The WMDP Benchmark. WMDP is a dataset of $3,\!668$ multiple-choice questions that serve as a proxy measure of hazardous knowledge in biosecurity, cybersecurity, and chemical security.

1 Introduction

Similar to other technologies, such as gene editing and nuclear energy, AI is dual-use—it can be leveraged for benefit and harm (Urbina et al., 2022). To address its dual-use risks, the White House Executive Order on Artificial Intelligence (White House, 2023) calls for investigation into the ability of AI to enable malicious actors in developing chemical, biological, radiological, nuclear, and cyber weapons. For instance, AI coding assistants may lower the barrier of entry for novices to conduct cyberattacks (Fang et al., 2024), potentially increasing the frequency of cyberattacks and the risk of catastrophe, especially if these attacks directed towards critical infrastructure, such as power grids (UK Cabinet Office, 2023). Likewise, AI assistants for biology could troubleshoot bottlenecks in biological weapons development, increasing the frequency of attempts to build a bioweapon and straining risk mitigation measures (Sandbrink, 2023). This has motivated government institutions and major AI labs to anticipate risk by designing evaluations for AI-aided biological threats (UK AI Safety Summit, 2023; Anthropic, 2023; OpenAI, 2024; Mouton et al., 2024; Phuong et al., 2024).

Unfortunately, current evaluations of hazardous capabilities do not provide a guide for mitigating malicious use risk. For example, developers evaluate whether models can build biological weapons end-to-end (Sandbrink, 2023) or hack well enough to exfiltrate their own weights (Shevlane et al., 2023), creating private, manual, and highly-specific evaluations. Because these evaluations test a small number of specific risk pathways, low performance on them does not guarantee that LLMs are secure across the broad distribution of malicious use risks. More importantly, such private benchmarking limits scientific inquiry towards measuring and reducing malicious use.

Developers also lack robust technical solutions to reduce malicious use in LLMs. The primary safeguard is training models to refuse harmful queries (Ouyang et al., 2022; Bai et al., 2022; Mazeika et al., 2024), but adversaries can deploy adversarial attacks (Wei et al., 2023; Zou et al., 2023b) to bypass models’ refusal training. Another proposal is to filter hazardous information from the pretraining data (Ngo et al., 2021), but adversaries may reintroduce this information through finetuning (Zhan et al., 2023; Qi et al., 2023; Pelrine et al., 2023). A promising approach for closed-source LLM providers is unlearning, directly removing hazardous knowledge before model serving (Figure 2). Unlearned models have higher inherent safety: even if they are jailbroken, unlearned models lack the hazardous knowledge necessary to enable malicious users (Hendrycks et al., 2021). However, research into unlearning hazardous knowledge is bottlenecked by the lack of a public benchmark.

To overcome both of these challenges, we introduce the Weapons of Mass Destruction Proxy Benchmark (WMDP), a benchmark of $3,\!668$ multiple-choice questions costing over $200K to develop (Figure 1). WMDP is a proxy measurement for hazardous knowledge in biosecurity (Section 3.2), cybersecurity (Section 3.3), and chemical security (Section 3.4). To design WMDP, academics and technical consultants created threat models for how LLMs might aid in the development of biological, cyber, and chemical attacks, and generated questions based on these threat models. We adopt a conservative stance towards including information in WMDP (Figure 3): we primarily include offensive knowledge, as unlearning defensive knowledge (e.g., biosafety protocols) may prevent benevolent use cases of LLMs. Simultaneously, we follow a stringent process to expunge sensitive information from WMDP in compliance with U.S. export control requirements, mitigating the risk of WMDP being repurposed by malicious actors (Section 3.5). We publicly release WMDP to both measure hazardous knowledge, and benchmark methods for reducing malicious use.

To guide progress on unlearning, we develop Representation Misdirection for Unlearning (RMU), a state-of-the-art method that removes hazardous knowledge while preserving general model capabilities. Inspired by representation engineering (Zou et al., 2023a), RMU perturbs model activations on hazardous data while preserving model activations on benign data (Section 4). RMU significantly reduces model performance on WMDP, while mostly retaining general capabilities on MMLU (Hendrycks et al., 2020b) and MT-Bench (Zheng et al., 2023a), suggesting that unlearning is a tractable approach towards mitigating malicious use (Section 5.2). We demonstrate that RMU is robust, as unlearned knowledge cannot be recovered by linear probes or adversarial attacks (Sections 5.2 and 5.3).

Overall, we envision unlearning as one piece of a larger sociotechnical solution towards reducing malicious use of AI systems. Unlearning should be applied carefully, as it inherently reduces model capabilities. Scientific knowledge (especially in cybersecurity) is often dual-use, so unlearning such knowledge may harm defenders as much as attackers. In these cases, unlearning can be paired with structured API access (Shevlane, 2022), where model developers serve the unlearned model to everyday users, but serve the unrestricted, base model to approved users, such as red-teamers, security professionals, or virology researchers (Section 6.2). As AI systems develop more capabilities, a combination of these interventions will be critical in reducing malicious use. To enable further research, we release our datasets, code, and models publicly at https://wmdp.ai.

2 Related Work

Evaluating risk from LLMs.

Recent work has highlighted safety concerns of language models, including generating falsehoods (Ji et al., 2023; Zhang et al., 2023), producing toxic content (Gehman et al., 2020; Deshpande et al., 2023; Pan et al., 2024), and deceiving humans (Park et al., 2023; Scheurer et al., 2023). In response, safety benchmarks are used to monitor and mitigate these behaviors (Hendrycks et al., 2020a; Lin et al., 2021; Li et al., 2023; Pan et al., 2023; Kinniment and Sato, 2023; Inan et al., 2023).

Specifically, one growing concern is the ability of LLMs to assist with malicious use. In particular, LLMs may aid actors in planning bioattacks (Sandbrink, 2023) and procuring pathogens (Gopal et al., 2023). Moreover, LLMs can assist users in synthesizing dangerous chemicals (Boiko et al., 2023) or conducting cyberattacks (Bhatt et al., 2023). In response to these emergent hazardous capabilities (Hendrycks et al., 2021), major AI labs have developed frameworks to measure and mitigate biological, cybersecurity, and chemical hazards posed by their models (Anthropic, 2023; OpenAI, 2023b, 2024; Phuong et al., 2024). Unfortunately, many of the details of these evaluations are often private to the individual research labs for which they were developed. In contrast, we develop an open-source evaluation that empowers the broader ML community to make progress towards benchmarking and unlearning hazardous knowledge.

Mitigating risk from LLMs.

Towards improving model safety, strategies such as input safety filtering (Inan et al., 2023) and learning from human preference data (Ziegler et al., 2020; Rafailov et al., 2023) have been developed; however, these methods can be vulnerable to jailbreaks (Wei et al., 2023; Chao et al., 2023; Yao et al., 2023a; Yuan et al., 2023) and adversarial attacks (Wallace et al., 2019; Guo et al., 2021; Jones et al., 2023; Zou et al., 2023b). To reduce inherent model risk, hazardous data can be removed prior to pretraining (Ngo et al., 2021), but having input into this process is inaccessible for most end users. Furthermore, models may be susceptible to subsequent harmful finetuning (Zhan et al., 2023; Yang et al., 2023) (Figure 2); as a result, and especially in the case of models that are accessed via API, additional automated methods that can be applied after finetuning—such as unlearning—may remove resulting hazards.

Machine unlearning.

Unlearning (Cao and Yang, 2015) originally gained traction as a response to privacy concerns in light of regulation (Council of European Union, 2014; CCPA, 2018), and most methods focused on erasing specific samples or facts (Golatkar et al., 2020; Liu et al., 2020; Meng et al., 2022; Jang et al., 2023; Pawelczyk et al., 2023) rather than entire domains. Goel et al. (2024) show existing unlearning methods struggle to remove knowledge without access to all relevant training data, a challenge RMU overcomes.

More recent methods erase broader concepts such as gender (Belrose et al., 2023), harmful behaviors (Yao et al., 2023b; Liu et al., 2024), or fictional universes (Eldan and Russinovich, 2023), but have not been proven to eliminate scientific knowledge which enables malicious use. Furthermore, most benchmarks for unlearning involve removing specific data samples (Google, 2023) or artificially chosen deletion sets (Choi and Na, 2023; Goel et al., 2023; Maini et al., 2024; Goel et al., 2024). In contrast, WMDP benchmarks on real-world information that can enable malicious use.

3 The WMDP Benchmark

We introduce the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of $3,\!668$ expert-written, multiple-choice questions in biosecurity (WMDP-Bio), cybersecurity (WMDP-Cyber), and chemistry (WMDP-Chem) costing over $200K to develop. The goal is to reduce question-answer (QA) accuracy on WMDP while maintaining performance on other benchmarks, such as MMLU (Hendrycks et al., 2020b) or MT-Bench (Zheng et al., 2023a). See Appendix A.1 for a breakdown of questions in WMDP and Appendix B.1 for a sample question.

WMDP is an automatic, public benchmark of hazardous capabilities that serves as a guide for risk mitigation (Section 3.1). We create questions by designing threat models for biosecurity (Section 3.2), cybersecurity (Section 3.3), and chemistry (Section 3.4). We also remove sensitive and export-controlled information from entering WMDP (Section 3.5). To further unlearning research beyond WMDP, we also provide additional unlearning benchmarks based on MMLU (Appendix C).

3.1 Design Choices for WMDP

Dataset form.

To create an automatic measure of hazardous capabilities that the broader research community can readily iterate on, we design WMDP as a dataset of four-choice multiple-choice questions. Multiple-choice is a common paradigm to test knowledge in language models (Hendrycks et al., 2020b; Rein et al., 2023).

Because WMDP measures knowledge of hazardous topics, models with a low score on WMDP likely lack the knowledge needed to help with malicious use. However, models with a high score on WMDP are not necessarily unsafe, as they may still lack the reasoning ability to combine the knowledge in the sequence of steps needed to create a weapon.

Dataset function.

WMDP should guide risk mitigation by enabling researchers to measure and reduce models’ hazardous capabilities. Because directly building a dataset of sensitive information would increase the attack capabilities of malicious actors (Esvelt, 2018; Lewis et al., 2019), we collect questions that approximate or correlate with the hazardous knowledge we wish to remove (Figure 3). In particular, we collect questions with knowledge that is a precursor, neighbor, or component of the hazardous knowledge we wish to remove. Moreover, we empirically demonstrate that models with lower performance on WMDP are less capable for malicious use (Section 5.4).

Examples of our dataset generation processes are detailed in Figure 4. In the left panel, research that aims to develop enhanced potential pandemic pathogens (ePPPs) is a precursor to developing novel viruses, so unlearning the former will also unlearn a large subset of the latter. In the center panel, there are topics in chemistry (e.g., procurement or synthesis) that contain questions with a wide variance in hazard level, so we approximate especially sensitive information by collecting questions near the boundary. In the right panel, a cyberweapon requires knowledge of several components (e.g., a payload, a trigger mechanism, and an infection mechanism), so excising knowledge of components will reduce hazards. Because some of the components may be dual-use, we generate questions for components that are primarily offensive in nature.

Dataset collection.

Our questions are written by academics and technical consultants in biosecurity, cybersecurity, and chemistry. We first generate threat models for each of these areas and then use the models to inform questions that an adversary might encounter when developing attack capabilities. To ensure quality, all of our questions were checked by at least two experts from different organizations.

3.2 Biosecurity Threat Model

In biosecurity, the malicious use threats that are increased by AI can be broadly categorized as expanding access to pre-existing threats (by lowering barriers to entry), and unlocking new areas of biology (by synthesizing new knowledge or accelerating in-silico modeling and experimentation) (Sandbrink, 2023).

We primarily focus on the development and dissemination of transmissible potential pandemic agents, such as influenza, smallpox, etc. While our dataset additionally includes some information about highly lethal non-transmissible bioweapons like anthrax, we believe the majority of emerging risk from biotechnology stems from advances in synthetic biology and bioengineering that increase access to, or modify, the design and development of transmissible agents (Esvelt, 2022).

A standard biotechnology risk chain can be seen in Figure 5. In this threat model, “ideation” involves actively planning for a biological attack; “design” involves retrieving blueprints for a hazardous agent, such as determining the DNA sequence; “build” consists of the protocols, reagents, and equipment necessary to create the threat; and “test” consists of measuring characteristics or properties of the pathogen of interest. By “learning” from these results and iterating after the test phase, the design can be revised until the threat is released (Nelson and Rose, 2023).

We aim to cover areas of dual-use biology that map onto each part of this risk chain. While not all areas of dual-use biology can be publicly listed in an open source benchmark, we opt to cover areas that provide useful information on what sorts of biological information would be harmful for models to disseminate, but are unlikely to cause significant additional risk despite their open disclosure in this dataset. We consulted with external biosecurity experts when formulating these areas, and describe additional considerations in Section A.2.

For “ideation,” we include areas of historical bioweapons and bioterrorism use (including methods, pathogen strains, and delivery systems used to mount bioterrorist attacks) that are publicly available online. We additionally include previously published information surrounding enhanced potential pandemic pathogens (ePPP) research, which involves investigating ways to make pre-existing pathogens more transmissible or virulent. Finally, we include what we believe are emerging areas of dual-use risk, such as viral vector engineering and research – while this research has significant benefits for medical treatments like gene therapy, we believe widespread access to information about viral vectors to the public is unlikely to provide significant value and could be actively harmful.

For the “build” portion of the design-build-test-learn (DBTL) cycle, we include knowledge on viral reverse genetics (protocols for reconstructing viruses), as well as niche areas of synthetic biology and bioengineering that may expand access to pathogen acquisition and release (including knowledge around circumventing DNA/RNA synthesis screening and export controls, as well as synthesizing biological agents). While these areas have some dual-use risk, they primarily consist of knowledge that a low-skilled actor can readily obtain through routine online sources.

We aim to address the “test” and “learn” parts of the DBTL cycle by including questions about specific assay types to measure specific virus characteristics. Finally, many aspects of “release” are included in questions formulated from the historical bioweapons and bioterrorism category; we did not include additional information about the release of potential pandemic pathogens to mitigate dual-use sensitive information risks.

3.3 Cybersecurity Threat Model

In cybersecurity, AI may aid malicious actors throughthe stages of a cyberattack: reconnaissance, weaponization, exploitation, and post-exploitation (Strom et al., 2020; Hutchins et al., 2011).

In the initial reconnaissance stage, attackers gather any available information about their target through both open-source and proprietary intelligence-gathering techniques. Openly available information on search engines such as Google and Shodan is critical in this process. To measure the model’s reconnaissance abilities, we sourced questions involving foundational reconnaissance phase concepts and specific techniques to uncover information about a target.

During the weaponization & vulnerability discovery stage, attackers use various methods to discover vulnerabilities within a target and craft exploits to take advantage of them. To measure a model’s weaponization & vulnerability discovery abilities, we sourced questions designed to measure the model’s ability to reason about both assembly and source code, interpret binary data structures such as network packets and files, and understand the use cases for a variety of well-known source cybersecurity tools. Very simple exploit generation tasks were included.

Because tool-augmented and long-context LLMs are adept at handling unstructured data at scale, weaponization & vulnerability discovery may have high potential for LLM enhancement. Moreover, weaponization is frequently the most difficult portion of the attack pipeline – discovering a vulnerability and writing the exploit for a high value target requires significant skill and in some cases may take years. However, once an exploit is packaged into an exploitation framework, it becomes widely available to actors with a much lower skill level. As a result, WMDP-Cyber places particular focus on question development at this stage.

During the exploitation stage, exploits designed to penetrate the target are launched to deliver a malicious payload – for instance, a software implant, a rootkit/bootkit, or simply a payload designed to crash the target device in the case of a DOS attack. Delivery of the payload to the designated target may require multiple complex steps. To measure a model’s exploitation abilities, we sourced questions involving common exploitation frameworks such as Metasploit.

Finally, after the payload is delivered, the desired post-exploitation activities are undertaken. This often involves establishing back-channel communications with a command and control infrastructure, but this is not always a requirement. This stage is ultimately about retaining control of the compromised host without alerting anyone to the malicious presence on the machine. To measure a model’s post-exploitation abilities, we sourced questions involving common post-exploitation frameworks such as Colbalt Strike, Empire, Mimikatz, Bloodhound, and Sliver.

3.4 Chemical Security Threat Model

In chemistry, similar to cybersecurity, AI can increase risk by aiding malicious actors through the stages of designing and deploying a chemical weapon. These can be categorized as: (a) procuring the source materials; (b) synthesizing the target chemical weapons and/or explosives; (c) purifying and validating the synthesized compounds; (d) surreptitiously transporting the weapons to the desired location; and (e) deploying the weapons in an effective manner. For a more detailed breakdown of the categories, see Appendix A.3.

Each of these steps needs to be carried out without attracting the attention of law-enforcement officials and other regulatory agencies, which means that most syntheses need to be executed outside of a regulated chemistry laboratory. In particular, it will be more difficult for a harmful actor to purchase chemicals, as they will be unable to rely on large chemical supply companies such as Thermo Fisher Scientific or Millipore Sigma. Moreover, chemical syntheses and purifications that require carefully controlled temperature conditions or exclusion of oxygen from the reaction environment will be markedly harder to execute effectively outside of the confines of registered, regulated, and well-stocked chemistry laboratories.

Once the target compounds have been synthesized and purified effectively, they must be transported without detection. Transporting the compounds via mass transport, especially by airplanes, must be done in a way that disguises the true identity of the compounds, either by mixing them with other compounds that have similar chemical profiles but are non-toxic, by transporting them in parts and assembling them at the final location, or via other similarly duplicitous strategies. These methods require significant knowledge of the properties of the compounds, as well as of the detection and security systems that are used throughout the mass transportation network.

Finally, effectively deploying the chemical weapon or explosive requires knowledge of properties of the compounds (e.g., the vapor pressure, solubility, or density) and how they operate. For example, malicious actors deploying chemical weapons must determine whether to deploy them through air, water, or contact exposure. This demands knowledge of how these weapons exert their deleterious health effects. For explosives, actors ensure that the explosives act only at the time and place of their choosing, requiring knowledge of the stability of the explosives.

3.5 Sensitive Information Mitigation

We implemented stringent procedures to ensure that no sensitive information is released in WMDP. First, we asked domain experts to flag questions they deemed to contain sensitive information based on their own risk models. Flagged questions were immediately excluded from the dataset. Aggregating opinions from discussions with academics and technical consultants, we identified that most concerns with sensitive information centered around WMDP-Bio and WMDP-Chem, so we took additional steps to mitigate sensitive knowledge in those categories. Specifically, we instituted a policy of “cross-checking” for WMDP-Bio and WMDP-Chem: on each question, two additional domain experts were tasked with determining whether the question constitutes sensitive information. Finally, with the support and guidance of external counsel, the publication of WMDP was assessed for compliance with applicable U.S. export control requirements, including with respect to the International Traffic in Arms Regulations (22 CFR Parts 120-130) (ITAR, 2024) and Export Administration Regulations (15 CFR Parts 730-774) (EAR, 2024).

4 RMU: Unlearning Inspired By Representation Engineering

We introduce Representation Misdirection for Unlearning (RMU), a finetuning method for unlearning hazardous knowledge (Algorithm 1). We outline the setup (Section 4.1) and explain our method (Section 4.2), with further detail in Appendix B.4 and B.5. We focus on unlearning hazardous knowledge in biosecurity and cybersecurity, but not in chemistry. While WMDP-Chem is a useful tool for hazard measurement, we are more uncertain if the hazard mitigation benefits of unlearning on WMDP-Chem outweigh the costs on general model capabilities.

4.1 Setup

We consider an autoregressive language model that accepts a prompt (e.g., “How can I synthesize anthrax?”) and returns a completion (e.g., “To synthesize anthrax, you need…”). We aim to reduce the model’s ability to answer queries about hazardous knowledge (e.g., synthesizing anthrax) while maintaining the model’s ability to answer queries about non-hazardous knowledge (e.g., culturing yeast). We operationalize this as reducing a model’s QA accuracy on WMDP while maintaining performance on general capabilities benchmarks, such as MMLU and MT-Bench.

In contrast to unlearning for copyright or privacy, we do not assume access to questions from WMDP. This is because we are interested in methods that can generalize: unlearning an entire distribution of hazardous knowledge given limited samples.

4.2 Method

Classically, language models are trained with a loss on their outputs (Vaswani et al., 2017; Devlin et al., 2018). On the other hand, mechanistic interpretability proposes editing models by intervening on individual neurons (Wang et al., 2022). In contrast to both these perspectives, we leverage the idea that model representations encode knowledge of the world (Meng et al., 2022) and that these representations may be manipulated to affect model behavior (Zou et al., 2023a; Ilharco et al., 2023; Turner et al., 2023). We design a two-part loss function with a forget loss and a retain loss; intuitively, the forget loss perturbs the model activations on hazardous data while the retain loss preserves its activations on benign data (Figure 7).

Forget loss.

Our goal is to degrade the model’s representations of hazardous knowledge. Our experiments suggest that increasing the norm of the model’s activations on hazardous data in earlier layers makes it difficult for later layers to process the activations, achieving our desiderata.

To calculate our forget loss, we assume access to $M_{\text{updated}}(\cdot)$ , the hidden states of the unlearned model at some layer $\ell$ and $M_{\text{frozen}}(\cdot)$ , the hidden states of the original, frozen model at some layer $\ell$ . Then, we compute $\mathbf{u}$ , a random unit vector with independent entries sampled uniformly at random from $[0,1)$ . Note that $\mathbf{u}$ is held fixed throughout training. Given a forget dataset $D_{\text{forget}}$ , we compute:

\mathcal{L}_{\text{forget}}=\mathbb{E}_{x_{f}\sim D_{\text{forget}}}\left[% \frac{1}{L_{f}}\sum_{\text{token }t\in x_{f}}\lVert M_{\text{updated}}(t)-c% \cdot\mathbf{u}\rVert_{2}^{2}\right]

where $L_{f}$ is the number of tokens in $x_{f}$ and $c$ is some hyperparameter that controls activation scaling.

Retain loss.

Our goal is to limit the amount of general capabilities lost from unlearning. Because our forget term is an $\ell^{2}$ loss on model activations, we regularize the model activations back to the original model’s activations with an $\ell^{2}$ penalty. Given the retain dataset $D_{\text{retain}}$ , we calculate the retain loss:

\mathcal{L}_{\text{retain}}=\mathbb{E}_{x_{r}\sim D_{\text{retain}}}\left[% \frac{1}{L_{r}}\sum_{\text{token }t\in x_{r}}\lVert M_{\text{updated}}(t)-M_{% \text{frozen}}(t)\rVert_{2}^{2}\right]

where $L_{r}$ is the number of tokens in $x_{r}$ .

Full loss.

The full loss (Figure 7) is a weighted combination of the forget loss and the retain loss:

\mathcal{L}=\mathcal{L}_{\text{forget}}+\alpha\cdot\mathcal{L}_{\text{retain}}.

RMU finetunes the model weights to minimize this loss. To unlearn multiple distributions of knowledge, we interleave the gradient updates (i.e., update model weights on the biosecurity distribution, then update on the cybersecurity distribution, then repeat). In practice, we find it sufficient to compute the loss only on layer $\ell$ and update gradients only on layers $\ell-2$ , $\ell-1$ , and $\ell$ . We leverage this observation to save memory and efficiently unlearn on larger LMs.

Forget and retain datasets.

To alter model activations on hazardous knowledge, we need to collect $D_{\text{forget}}$ , an unlearning distribution which approximates WMDP. To collect $D_{\text{forget}}$ for biosecurity, we collect a corpus of relevant papers from PubMed used to generate questions in WMDP-Bio (Section A.4). To collect $D_{\text{forget}}$ for cybersecurity, we conduct an extensive crawl of GitHub for documents associated with the topics in WMDP-Cyber, and filter the contents to include only the most relevant passages to WMDP-Cyber (Section A.5).

Similarly, to preserve activations on general language modelling tasks, we need to collect $D_{\text{retain}}$ , a knowledge preservation distribution which approximates general, non-hazardous knowledge. For these, we collected subject-specific retain sets detailed in Sections A.4 and A.5. However, we find in practice that RMU is more performant when $D_{\text{retain}}$ has qualitatively distinct content from $D_{\text{forget}}$ , so as not to relearn the unlearned knowledge. Thus, we set $D_{\text{retain}}$ to be Wikitext (Merity et al., 2016). We release the unused subject-specific retain sets for WMDP-Bio and WMDP-Cyber publicly, to guide future unlearning methods that can more effectively use these corpora.

1:Input: Updated model

M_{\text{updated}}

, frozen model

M_{\text{frozen}}

, forget dataset

D_{\text{forget}}

, retain dataset

D_{\text{retain}}

\triangleright

Model returns layer

\ell

’s activations

2:function RMU(

D_{\text{forget}}

D_{\text{retain}}

c

\alpha

)

3: Sample unit vector

\mathbf{u}

with independent entries drawn uniformly at random from

[0,1)

4: for data points

x_{\text{forget}}\sim D_{\text{forget}},x_{\text{retain}}\sim D_{\text{retain}}

5: Set

\mathcal{L}_{\text{forget}}=\frac{1}{L}\sum_{\,\text{token }t\in x_{\text{% forget}}}\lVert M_{\text{updated}}(t)-c\cdot\mathbf{u}\rVert_{2}^{2}

where

x_{\text{forget}}

L

tokens long

6: Set

\mathcal{L}_{\text{retain}}=\frac{1}{L}\sum_{\,\text{token }t\in x_{\text{% retain}}}\lVert M_{\text{updated}}(t)-M_{\text{frozen}}(t)\rVert_{2}^{2}

where

x_{\text{retain}}

L

tokens long

7: Update weights of

M_{\text{updated}}

using

\mathcal{L}=\mathcal{L}_{\text{forget}}+\alpha\cdot\mathcal{L}_{\text{retain}}

\triangleright

Loss on model activations

8: end for

9: return

M_{\text{updated}}

10:end function

Algorithm 1 RMU Pseudocode

5 Experimental Results

We examine the performance of RMU and other unlearning methods. We describe the experimental setup (Section 5.1) and provide quantitative (Section 5.2) and robustness (Section 5.3) evaluations. We also check if unlearning on WMDP generalizes to more hazardous information (Section 5.4). Finally, we report plots for how RMU scales activations on hazardous and benign data in Appendix B.5. RMU markedly improves upon existing baselines, but future work is necessary to improve the precision of unlearning hazardous knowledge while fully maintaining general capabilities.

Model	WMDP ( $\downarrow$ )		MMLU ( $\uparrow$ )	MT-Bench ( $\uparrow$ )
Model	Bio	Cyber	MMLU ( $\uparrow$ )	MT-Bench ( $\uparrow$ )
zephyr-7b	$63.7$	$44.0$	$58.1$	$7.33$
\cdashline1-5 $+$ LLMU	$59.5$	$39.5$	$44.7$	$1.00$
$+$ SCRUB	$43.8$	$39.3$	$51.2$	$1.43$
$+$ SSD	$50.2$	$35.0$	$40.7$	$5.48$
$+$ RMU (ours)	$\mathbf{31.2}$	$\mathbf{28.2}$	$\mathbf{57.1}$	$\mathbf{7.10}$
Yi-34b	$75.3$	$49.7$	$72.6$	$7.65$
\cdashline1-5 $+$ RMU (ours)	$30.7$	$29.0$	$70.6$	$7.59$
Mixtral-8x7B	$74.8$	$52.0$	$68.2$	$8.30$
\cdashline1-5 $+$ RMU (ours)	$34.0$	$30.8$	$67.1$	$8.17$

	Category	Count
Bio	Dual-use Virology	228
	Bioweapons & Bioterrorism	197
	Reverse Genetics & Easy Editing	252
	Enhanced Potential Pandemic Pathogens	233
	Viral Vector Research	228
	Expanding Access	135
	Total	1,273
Cyber	Background Knowledge	271
	Reconnaissance	20
	Weaponization & Vulnerability Discovery - Assembly Review	283
	Weaponization & Vulnerability Discovery - Function Review	300
	Weaponization & Vulnerability Discovery - Packet Dissection	298
	Weaponization & Vulnerability Discovery - Other	361
	Exploitation	272
	Post-Exploitation	182
	Total	1,987
Chem	General Knowledge	127
	Synthesis	78
	Sourcing / Procurement	41
	Purification	19
	Analysis / Verification	21
	Deployment Mechanisms	65
	Bypass Mechanisms	15
	Miscellaneous	42
	Total	408

Model	Method	WMDP $(\downarrow)$			MMLU $(\uparrow)$					MT-Bench $(\uparrow)$
Model	Method	Bio	Cyber	Chem	College Bio	Virology	College CS	Cybersec	All	MT-Bench $(\uparrow)$
zephyr-7b	Base	63.7	44.0	45.8	68.1	52.4	50.0	65.0	58.1	7.33
\cdashline2-11			LLMU	59.5	39.5	41.4	54.2	37.4	43.0	53.0	44.7	1.00
	SCRUB	43.8	39.3	40.4	53.5	40.3	48.0	62.0	51.2	1.43
	SSD	50.2	35.0	33.8	46.5	38.0	35.0	52.0	40.7	5.48
	RMU (ours)	31.2	28.2	45.8	63.2	25.9	49.0	45.0	57.1	7.10
Yi-34b	Base	75.3	49.7	58.6	88.9	57.2	63.0	84.0	72.6	7.65
\cdashline2-11			RMU (ours)	30.7	29.0	55.4	84.0	22.3	57.0	46.0	70.6	7.59
Mixtral-8x7B	Base	74.8	52.0	55.2	82.6	50.0	64.0	80.0	68.2	8.30
\cdashline2-11			RMU (ours)	34.0	30.8	54.7	81.3	34.3	67.0	58.0	67.1	8.17
GPT-4	Base	82.2	55.3	64.7	93.9	58.2	69.0	84.5	83.4	9.13

Category	Forget		Retain		MMLU (Full)
	Base	RMU	Base	RMU	Base	RMU
Physics	38.8	27.0	34.6	29.2	58.6	57.1
Law	56.7	27.8	71.3	37.0	58.6	54.5
Economics	60.2	27.3	45.6	41.2	58.6	55.0

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

Abstract

1 Introduction

2 Related Work

Evaluating risk from LLMs.

Mitigating risk from LLMs.

Machine unlearning.

3 The WMDP Benchmark

3.1 Design Choices for WMDP

Dataset form.

Dataset function.

Dataset collection.

3.2 Biosecurity Threat Model

3.3 Cybersecurity Threat Model

3.4 Chemical Security Threat Model

3.5 Sensitive Information Mitigation

4 RMU: Unlearning Inspired By Representation Engineering

4.1 Setup

4.2 Method

Forget loss.

Retain loss.

Full loss.

Forget and retain datasets.

5 Experimental Results

5.1 Setup

Benchmarks.

Models.

Baselines.

5.2 Quantitative Evaluation

Forget performance.

Retain performance.

5.3 Robustness Evaluation

5.4 Generalization of WMDP to Hazardous Knowledge

6 Discussion

6.1 How WMDP Mitigates Risk

6.2 Structured API Access

7 Conclusion

Acknowledgments

References

Appendix A Dataset

A.1 Dataset Breakdown

A.2 Additional Considerations for WMDP-Bio

A.3 Additional Considerations for WMDP-Chem

A.4 Bio Corpora

A.5 Cyber Corpora

Appendix B Experiments

B.1 Zero-Shot QA Format

B.2 MT-Bench

B.3 Robustness Evaluation

B.3.1 RMU Unlearned Model

B.3.2 Base Model

B.4 Updates to RMU

B.5 How RMU manipulates representations

B.6 Generalization of RMU

B.7 Baselines

B.7.1 LLMU

B.7.2 SCRUB

B.7.3 SSD

B.7.4 RMU

Appendix C MMLU Subset Unlearning Benchmark

Appendix D Broader Impacts of WMDP

D.1 Limitations

Appendix E X-Risk Sheet

E.1 Long-Term Impact on Advanced AI Systems

E.2 Safety-Capabilities Balance

E.3 Elaborations and Other Considerations

The WMDP Benchmark: Measuring and Reducing
Malicious Use With Unlearning