Harmfully Manipulated Images Matter in Multimodal Misinformation Detection

Bing Wang, Shengsheng Wang, Changchun Li, Renchu Guan, Ximing Li

Abstract: Nowadays, misinformation is widely spreading over various social media platforms and causes extremely negative impacts on society. To combat this issue, automatically identifying misinformation, especially those containing multimodal content, has attracted growing attention from the academic and industrial communities, and induced an active research topic named Multimodal Misinformation Detection (MMD). Typically, existing MMD methods capture the semantic correlation and inconsistency between multiple modalities, but neglect some potential clues in multimodal content. Recent studies suggest that manipulated traces of the images in articles are non-trivial clues for detecting misinformation. Meanwhile, we find that the underlying intentions behind the manipulation, e.g., harmful and harmless, also matter in MMD. Accordingly, in this work, we propose to detect misinformation by learning manipulation features that indicate whether the image has been manipulated, as well as intention features regarding the harmful and harmless intentions of the manipulation. Unfortunately, the manipulation and intention labels that make these features discriminative are unknown. To overcome the problem, we propose two weakly supervised signals as alternatives by introducing additional datasets on image manipulation detection and formulating two classification tasks as positive and unlabeled learning problems. Based on these ideas, we propose a novel MMD method, namely Harmfully Manipulated Images Matter in MMD (MANI-M$^3$D). Extensive experiments across three benchmark datasets can demonstrate that \baby can consistently improve the performance of any MMD baselines.

Primary Subject Area: [Engagement] Emotional and Social Signals

Secondary Subject Area: [Experience] Multimedia Applications, [Content] Multimodal Fusion

Relevance To Conference: This work focuses on Multimodal Misinformation Detection (MMD), which aims to identify misinformation consisting of multimodal content, such as image and text. Some recent studies suggest that if an article contains manipulated traces in its visual content, it is more likely to be identified as misinformation. Meanwhile, It is also non-trivial to delve into understanding the underlying intention behind the image manipulation. Motivated by these ideas, we propose a novel MMD framework using PU learning techniques. Extensive experimental results demonstrate the effectiveness of our model.

Submission Number: 3377