Information Aggregate and Sentiment Enhance Network to Handle Missing Modalities for Multimodal Sentiment Analysis

Liang Shi, Fuyong Xu, Ru Wang, Yongqing Wei, Guangjin Wang, Bao Wang, Peiyu Liu

Abstract: Multimodal Sentiment Analysis(MSA) mostly based on the assumption that all modalities are available. However, this assumption does not always hold in practice, and the performance of most multimodal sentiment analysis models can suffer significant degradation when some modalities are missing. To this end, we propose an Information Aggregation and Sentiment Enhance network(IASE) to handle missing modalities. Specifically, IASE models multimodal data as a bipartite graph structure that reduces the impact of missing modalities by aggregating complementary information to update out the representation of missing modalities. Aggregating samples of different sentiments leads to a decrease of sentiment intensity. The sentiment enhancement module is designed to enhance the sentiment intensity. Extensive experiments on several datasets show significant improvements of our method compared to several baselines.