Multimodal transformer with adaptive modality weighting for multimodal sentiment analysis

Yifeng Wang, Jiahao He, Di Wang, Quan Wang, Bo Wan, Xuemei Luo

Abstract: Highlights•Novel multimodal adaptive weight matrix enables accurate sentiment analysis by considering unique contributions of each modality.•Multimodal attention mechanism addresses over-focusing on intra-modality attention.•Multiple Softmax functions compute attention weights, optimizing self-attention of concatenated sequences.•Enhanced sentiment analysis through unique contributions of each modality.•Outperforms existing state-of-the-art models in extensive experiments.