AI-Generated Images Introduce Invisible Relevance Bias to Text-Image Retrieval

Shicheng Xu, Danyang Hou, Liang Pang, Jingcheng Deng, Jun Xu, Huawei Shen, Xueqi Cheng

Keywords: Bias, Text-Image Retrieval, AIGC

TL;DR: This paper is dedicated to exploring the impact of images generated by foundation models on search.

Abstract:

With the application of foundation models, internet is increasingly inundated with AI-generated content (AIGC), causing both real and AI-generated content indexed in corpus for search. This paper explores the impact of AI-generated images on text-image search in this scenario. Firstly, we construct a benchmark consisting of both real and AI-generated images for this study. In this benchmark, AI-generated images possess visual semantics sufficiently similar to real images. Experiments on this benchmark reveal that text-image retrieval models tend to rank the AI-generated images higher than the real images, even though the AI-generated images do not exhibit more visually relevant semantics to the queries than real images. We call this bias as invisible relevance bias. This bias is detected across retrieval models with different training data and architectures. Further exploration reveals that mixing AI-generated images into the training data of retrieval models exacerbates the invisible relevance bias. These problems cause a vicious cycle in which AI-generated images have a higher chance of exposing from massive data, which makes them more likely to be mixed into the
training of retrieval models and such training makes the invisible relevance bias more and more serious. Findings in this paper reveal the potential impact of AI-generated images on text-image retrieval and have implications for further research.

Submission Number: 48