An Auxiliary Modality Based Text-Image Matching Methodology for Fake News Detection

Ying Guo, Bingxin Li, Hong Ge, Chong Di

Abstract: Owing to the national network “clearing up” action, it has become increasingly important to detect false information by the use of deep learning technology. As social networks gradually presents a multimodal property, many scholars have devoted to multimodal fake news detection. However, the current multimodal achievements mainly focus on the fusion modeling between texts and images, while their consistencies are still in their infancy. This paper concentrates on the issue of how to extract effective features from texts and images, how to match modes in a more precise way, and subsequently proposes a novel fake news detection method. Especially, the models of Bert, Vgg, and Optical Character Recognition (OCR) are respectively adopted to reflect the textual features, the visual counterparts, as well as the corresponding embedded contents in the attachment. The overall model framework consists of four components: one fusion module and three matching modules, where the former one joints text and image features, and the latter three computes the corresponding similarities among textual, visual, and auxiliary modalities. Aligning them with different weights, and connecting them with a classifier, whether the news is fake or real can thus emerge. Comparative experiments embody the effectiveness of our model, which can reach 88.1%’s accuracy on the Chinese Weibo dataset and 91.7%’s accuracy on the English Twitter dataset.