Event-Based Multi-Modal Fusion for Online Misinformation Detection in High-Impact Events

Javad Rajabi, Sunday Okechukwu, Ahmad Mousavi, Roberto Corizzo, Charles C. Cavalcante, Zois Boukouvalas

Abstract: Social media platforms are pivotal in information dissemination but also contribute to the rapid spread of misinformation, especially during high-impact events like natural disasters, terrorist attacks, and political unrest. While recent advances in multi-modal learning have enhanced misinformation detection by integrating features from various modalities (e.g., text, images), certain areas remain under-explored, particularly the use of event-based multi-modal data. This paper introduces a novel approach to misinformation detection on social media using an event-based multi-modal learning framework. Our method extends beyond traditional techniques by employing latent variable modeling to capture non-linear associations in event-based multi-modal data and to generate joint features between events for classification. This approach enhances misinformation detection and enables the contextual understanding of terms across different events. We provide a detailed analysis of our dataset preparation, methodology, and results, demonstrating the effectiveness of our framework on a widely-used dataset of tweets from high-impact events. The paper concludes with insights into potential enhancements and future directions in multi-modal misinformation detection.