Re: Data Poisoning Attacks Against Multimodal Encoders
Abstract: Multimodal models, which leverage both visual and linguistic modalities, have gained increasing attention in recent years. However, these models are often trained on large-scale unlabeled datasets, which expose them to the risk of data poisoning attacks. An adversary can manipulate the training data to induce malicious behaviors in the model under certain conditions. Yang et al. (2023) recently conducted a study on the susceptibility of multimodal models to poisoning attacks. They introduced three types of poisoning attacks targeted at multimodal models, along with two potential defenses. In this work, we replicate all three attack strategies. However, we observed that the effectiveness of the attack depends on the poisoning rate in relation to the quantity of samples in the targeted class, a factor that can potentially reduce the efficiency of the attack. Additionally, we replicated the ablation study, verified the consistency of their claims, and provided further experimentation to test them. Regarding the proposed defenses, we reproduced them and explained a flaw in the first defense. Furthermore, we propose a more practical setting for the second defense.
Submission Length: Long submission (more than 12 pages of main content)
Assigned Action Editor: ~W_Ronny_Huang1
Submission Number: 2232
Loading