Multimodal Sentiment Analysis with Common-sense Modulation

Anonymous

Abstract: Our world is inherently multimodal and recent work highlights the importance of machine learning models leveraging multiple streams of information in making decisions. Multimodal sentiment analysis has been an active area of research that requires models to take advantage of the linguistic, acoustic, and visual signals available in an utterance. However, most current models do not take into account any social common-sense knowledge which is crucial in how we perceive sentiment in a conversation. To address that, in this paper, we aim to influence or modulate modality representations with common-sense knowledge obtained from a generative social common-sense knowledge base. We provide a novel way to modulate the linguistic, acoustic, and visual features corresponding to an utterance by scaling and shifting these representations. We use the knowledge base to obtain knowledge latent representations for an utterance corresponding to different states of the speaker such as the intent and the reaction, and we use it to shift and scale the three modalities. Our experiments on popular multimodal sentiment analysis benchmark datasets show that our proposed method is on par and often surpasses the current state-of-the art models.