Exploring Image-Text Discrepancy for Universal Fake Image Detection

Daichi Zhang, Tong Zhang, Jianmin Bao, Yixin Cheng, Shiming Ge, Sabine Susstrunk

Keywords: vision-language model, image-text discrepancy, fake image detection

Abstract:

With the rapid development of generative models, detecting generated images to prevent their malicious use has become a critical issue recently. Existing methods frame this challenge as a binary image classification task. However, such methods focus only on visual space, yielding trained detectors susceptible to overfitting specific image patterns and incapable of generalizing to unseen models. In this paper, we address this issue from a multi-modal perspective and find that fake images exhibit more distinct discrepancies with corresponding captions compared to real images. Upon this observation, we propose to leverage the \textbf{I}mage-\textbf{T}ext \textbf{D}iscrepanc\textbf{y}~(\textbf{TIDY}) in joint visual-language space for \textit{universal fake image detection}. Specifically, we first measure the distance of the images and corresponding captions in the latent spaces of CLIP, and then tune an MLP head to perform the usual detection task. Since there usually exists local artifacts in fake images, we further propose a global-to-local discrepancy scheme that first explores the discrepancy on the whole image and then each semantic object described in the caption, which can explore more fine-grained local semantic clues. Extensive experiments demonstrate the superiority of our method against other state-of-the-art competitors with impressive generalization and robustness on various recent generative models.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3812