Multimodal Content Veracity Assessment with Bidirectional Transformers and Self-Attention-based Bi-GRU Networks

Jenq-Haur Wang, Mehdi Norouzi, Shu Ming Tsai

Abstract: Sharing information on social media has become a part of people's daily lives. However, without centralized management on user generated contents, increasing amount of rumors and misinformation are being spread in social media. On the one hand, most of the social platforms debunk rumors with manual verification by fact checking organizations, which are very inefficient. On the other hand, since it's common for rumors to contain inconsistent images and texts, it would be useful if we could compare the semantics between multimodal contents in the same post for rumor detection. In this paper, we propose to check multimodal content consistency with transformers and self-attention-based Bi-GRU networks for rumor detection. Firstly, image semantic contents are extracted by image captioning module to generate captions. Then, the generated captions are semantically compared with texts using transformers for veracity assessment. Finally, Multi-cell bi-directional Recurrent Neural Networks (Bi-RNNs) with self-attention mechanism are used to find word dependency and learn the most important features for rumor detection. From the experimental results on tweets, the best F1-score of 0.92 can be obtained for our proposed approach to multimodal veracity assessment. This shows the potential of our proposed method in rumor detection. Further investigation is needed to verify the performance using different multimodal features.