Backdoor Attacks on Bimodal Salient Object Detection with RGB-Thermal Data

Wen Yin, Bin Benjamin Zhu, Yulai Xie, Pan Zhou, Dan Feng

Abstract: RGB-Thermal salient object detection (RGBT-SOD) plays a critical role in complex scene recognition fields such as autonomous driving, yet security research in this area remains limited. This paper introduces the first backdoor attack targeting RGBT-SOD, generating saliency maps on triggered inputs that depict non-existent salient objects chosen by the attacker, or designate no salient region (all black pixels) or the entire image as a salient region (all white pixels). We uncover that triggers possess an influence range for generating non-existent salient objects, supported by a theoretical approximation provided in this study. Extensive experimental evaluations validate the efficacy of our attack in both digital domain and physical-world scenarios. Notably, our dual-modality backdoor attack achieves an Attack Success Rate (ASR) of 86.72% with only 5 pairs of images in model training. Despite exploring potential countermeasures, we find them ineffective in thwarting our attacks, underscoring the urgent need for robust defenses against sophisticated backdoor attacks in RGBT-SOD systems.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: In this work, we introduce the first backdoor attack on SOD, specifically targeting bimodal RGBT-SOD models. We devise a bimodal trigger, manifested as a heating device, which exhibits distinct characteristics from the environmental background in both RGB and thermal infrared modes. This trigger can induce backdoor responses from both modalities of the bimodal SOD model. We conduct an extensive experimental evaluation of our proposed backdoor attack on bimodal SOD models, utilizing three RGBT-SOD models and three publicly available RGBT datasets. Our real-world attack achieves an ASR of 92.00% from various viewing angles. Notably, Our attack can be successful by triggering both modalities simultaneously or each modality individually. Our study reveals the threat of backdoor attacks on RGBT-SOD as well as general SOD models and calls for developing effective countermeasures to thwart them.

Submission Number: 2544