Alignment-aware Data Selection for Unlearning in Contrastive Vision-Language Models

Dongjun Hwang, Yejin Kim, Beomyun Kwon, Junsuk Choe

Keywords: Machine Unlearning, Data Selection, Contrastive Vision-Language Models

TL;DR: ALISE selects forget samples based on alignment to improve unlearning performance in contrastive VLMs.

Abstract: Recent advances in contrastive vision-language models have increased the need to selectively remove knowledge of specific data, drawing attention to _machine unlearning_. In this paper, we observe that unlearning performance in contrastive VLMs largely depends on the composition of the forget set. Based on this insight, we propose **ALISE**, a data selection framework that measures each forget sample’s alignment with both the retain set and the full forget set, and selects samples accordingly. Extensive experiments across diverse downstream applications demonstrate that ALISE facilitates removing target knowledge in contrastive VLMs while preserving model utility.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 36