In-Context Unlearning: Language Models as Few Shot Unlearners

Martin Pawelczyk, Seth Neel, Himabindu Lakkaraju

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Machine unlearning, In-context unlearning, Right to be forgotten, Approximate data deletion

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We suggest a new class of unlearning problems that is relevant when LLMs are trained through an API where no parameter access is available. We then suggest the first solution to this problem called In-Context Unlearning (ICUL).

Abstract: Machine unlearning has garnered increased attention within regulatory contexts, driven by the need to comply with the "Right to be Forgotten''. However, achieving precise unlearning is computationally infeasible for large models, particularly when dealing with large language models (LLMs). To this end, several algorithms which approximate the removal of training data without retraining the model have been proposed which rely on gradient ascent based model updates. In this work, we propose a new class of unlearning methods called "In-Context Unlearning'' suitable for LLMs by providing inputs in context and without having to update model parameters. To unlearn a particular training instance, we provide the instance alongside a different label and additional correctly labelled instances as inputs to the LLM at inference time. Our experimental results across various text classification tasks demonstrate that these contexts effectively remove specific information from the training set while maintaining performance levels that are competitive with state-of-the-art unlearning methods that require access to the LLM parameters.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7822