Hidden Logos in Web-Scale Data Disrupt Large Vision Language Models
Keywords: Spurious Correlations, Bias, Logos
Abstract: Vision-Language Models are trained on very large, minimally curated image datasets that contain many spurious correlations between categories and visual patterns. This causes VLMs to learn shortcuts, e.g., between smiling and gender. Although logos are ubiquitous in VLM training data and are a potential source of such shortcuts, there is very limited study of this issue. Prior work pointed out that logos may indeed cause such problems, but the analysis was limited to a single text-based logo. In this paper, we undertake a broad study of logos in VLM training data and their potential to insert "hidden" spurious correlations into VLMs. We construct a new logo dataset, CC12M-LogoBank, propose an algorithm that uncovers spurious logos affecting a given VLM prediction task, and test it on several representative tasks: person attribute classification, object classification, and harmful content detection. Our key finding is that some logos indeed lead to spurious incorrect predictions, for example, adding the Adidas logo to a photo of a person causes a model classify the person as "greedy". Furthermore, we argue that the uncovered logos can be seen as effective attacks against foundational models; for example, an attacker could place a spurious logo on harmful content, causing the model to misclassify it as harmless. This threat is alarming considering the simplicity of logo attacks, increasing the attack surface of VLM models. As a defense, we explore two effective yet simple mitigation strategies that seamlessly integrate with zero-shot inference of foundation models.
Supplementary Material: zip
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3236
Loading