Segment Anything Meets Universal Adversarial Perturbation
Keywords: Universal Adversarial Perturbation, Adversarial Robustness, Segment Anything
TL;DR: Universal Adversarial Attack on Segment Anything
Abstract: As Segment Anything Model (SAM) becomes a popular foundation model in computer vision, its adversarial robustness has become a concern that cannot be ignored. In this work, we investigate whether it is possible to attack SAM with image-agnostic Universal Adversarial Perturbation (UAP). In other words, we seek a single perturbation that can fool the SAM to predict invalid masks for most images. We conduct a preliminary investigation and find that universal adversarial attack on SAM is a non-trivial task under the traditional supervised paradigm by focusing on destroying the features in the images. Considering its image-agnostic property, the UAP itself is expected to have independent features. Motivated by this rationale, we propose a novel self-supervised contrastive learning (CL) framework for crafting a UAP. Specifically, we treat the UAP as an anchor image with independent features, with random images and UAP augmented with random images set to negative and positive samples. Extensive experiments verify the effectiveness of our method. Another merit of our proposed method is that the infoNCE attack loss calculated on the embedded feature space attack requires no access to the SAM mask decoder, which makes our universal attack method prompt-agnostic and thus further enhances its flexibility.
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9403
Loading