AN INFORMATION THEORETIC EVALUATION METRIC FOR STRONG UNLEARNING

Dongjae Jeon, Wonje Jeung, Taeheon Kim, Albert No, Jonghyun Choi

Keywords: Strong unlearning, information-theoretic metrics, Information Difference Index, residual information, evaluation metric

TL;DR: We introduce IDI to overcome black-box limitations in strong unlearning evaluation, validated through experiments and benchmarked with COLA.

Abstract:

Machine unlearning (MU) aims to remove the influence of specific data from trained models, addressing privacy concerns and ensuring compliance with regulations such as the "right to be forgotten."
Evaluating strong unlearning, where the unlearned model is indistinguishable from one retrained without the forgetting data, remains a significant challenge in deep neural networks (DNNs).
Common black-box metrics, such as variants of membership inference attacks and accuracy comparisons, primarily assess model outputs but often fail to capture residual information in intermediate layers.
To bridge this gap, we introduce the Information Difference Index (IDI), a novel white-box metric inspired by information theory.
IDI quantifies retained information in intermediate features by measuring mutual information between those features and the labels to be forgotten, offering a more comprehensive assessment of unlearning efficacy.
Our experiments demonstrate that IDI effectively measures the degree of unlearning across various datasets and architectures,
providing a reliable tool for evaluating strong unlearning in DNNs.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10404