Graph-based Fine-grained Multimodal Attention Mechanism for Sentiment Analysis

Anonymous

Abstract: Multimodal sentiment analysis is a popular research area in natural language processing. Mainstream multimodal learning models barely consider that the visual and acoustic behaviors often have a much higher temporal frequency than words. Therefore, these models lack the representation capability to accurately model multimodal interactions. In this paper, we propose an attachment called Graph-based Fine-grained Multimodal Attention Mechanism (GFMAM), which can utilize the multimodal information from different subspaces to achieve accurate multimodal interactions. Firstly, the attachment further splits the information of every modality into multiple subspaces.Then, the fine-grained multimodal information from different subspaces is converted into multimodal interaction graphs dominant by the language modality. The multimodal interaction graph can capture significant interactions among multiple modalities at the subspace level.Finally, the information of nonverbal modalities is additionally added to compensate for the loss of continuity caused by the splitting operation. Embedding GFMAM into BERT, we propose a new model called GFMAM-BERT that can directly accept nonverbal modalities in addition to language modality. We conducted experiments on both publicly available multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results demonstrate that GFMAM-BERT exceeds the state-of-the-art models. Moreover, the proposed model outperforms humans on most metrics on the CMU-MOSI dataset.