Learning and Forgetting Unsafe Examples in Large Language Models
Primary Area: societal considerations including fairness, safety, privacy
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Large language models; Safety alignment; Neural networks forgetting
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: As the number of large language models (LLMs) available to the public grows, there is a pressing need to understand the safety implications associated with these models learning from third-party custom finetuning data. We explore the behavior of LLMs finetuned on unsafe content, represented by datasets that contain biases, toxicity, and harmfulness, finding that while LLMs can readily learn this unsafe content, they also tend to forget it when subsequently finetuned on safer content. Drawing inspiration from this forgetting behavior, we introduce the ``\ff{}'' algorithm, which filters unsafe data based on how strong the model's forgetting signal is for that data. We find that the \ff{} algorithm outperforms alternative strategies like replay and moral self-correction in curbing LLMs' ability to assimilate unsafe content during custom finetuning, e.g. 75\% lower than not applying any safety measures and 62\% lower than using self-correction in toxicity score.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8876
Loading