Polarity-aware attention network for image sentiment analysis

Qiming Yan, Yubao Sun, Shaojing Fan, Liling Zhao

Abstract: Image sentiment analysis aims to employ a computational model to automatically discover the implied emotions from the underlying image, which are crucial in many practical applications. The psychological finding demonstrates that the emotional content is ordinarily involved in some informative regions. In fact, these informative regions of visual images also convey different emotional polarities and intensities. The emotion conveyed by the whole image can be regard as the combined effect of the positive-polarity emotional regions and the negative-polarity emotional regions. Motivated by this psychological prior knowledge, we propose a new polarity-aware attention network for image sentiment analysis in an end-to-end manner. Specifically, the proposed network is composed of a sentimental feature extraction backbone, a polarity-aware attention module and a fused classification module. The backbone is used to extract the global contextual features. The polarity-aware attention module not only attends to positive and negative emotion regions by predicting polarity-aware attention maps, but also estimates their polarity intensities. The fused classification module integrates the output of the first two modules for the final image sentiment prediction. A compound loss function is designed to guide network learning using the weakly-supervised manner and the distributed label smooth learning method. We validate our method on multiple benchmarks and the experimental results demonstrate that our method can obtain superior performance over several state-of-the-art methods.