CaRE: Continual Real-time Unlearning with Ensured Preservation of LLM Knowledge

Seyun Bae, Seokhan Lee, Eunho Yang

Keywords: Real-time unlearning, continual unlearning, large language models

TL;DR: We present a novel method for achieving effective continual unlearning in large language models in real-time.

Abstract:

As concerns grow over the issue of large language models (LLMs) inadvertently internalizing sensitive or erroneous information, unlearning—the selective removal of undesired knowledge—has been drawing an increasing amount of attention.
Existing approaches to unlearning fail to account for scenarios requiring immediate processing of knowledge removal requests, leaving services that rely on LLMs vulnerable to prolonged leakage of sensitive information while the process of unlearning is underway.
Moreover, when such requests occur not just once, but continuously over the period of deployment, current methods cause LLMs to suffer increasingly degraded utility performance with the processing of each request.
To address these issues, we propose **C**ontinu**a**l **R**eal-time Unlearning with **E**nsured Preservation of LLM Knowledge (**CaRE**). Prior to LLM deployment, we train an unlearning sentence embedder with a synthetically generated dataset designed to enable the formation of sharp decision boundaries for determining whether a given input query corresponds to any forget requests in the database. At inference, an embedding is generated for the input query and compared with the embedding of each forget request using a distance metric and the maximum score is compared to a threshold which is used to decide whether to answer the query or to refuse. Since our method does not modify any weights of the language model, it avoids catastrophic forgetting and is able to achieve near perfect knowledge preservation after an arbitrary number of updates. Our experiments on four benchmarks demonstrate that **CaRE** achieves a superior balance of forgetting and knowledge preservation over all existing methods in the continual setting while also being the only method capable of processing forget requests in real-time.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 10548