Generating Emotion Descriptions for Fine Art Paintings Via Multiple Painting Representations

Yue Lu, Chao Guo, Xingyuan Dai, Fei-Yue Wang

Abstract: The task of generating emotion descriptions for fine art paintings using machine learning is gaining increasing attention. However, captioning the emotions depicted in paintings is challenging due to the artistic and subtle nature of the relied-upon visual clues. Previous studies on painting emotion captioning mainly focus on content-oriented semantic features, resulting in limited performance. Recognizing that facial expressions and body language can reflect human emotions, we propose a novel painting emotion captioning model that incorporates two additional features: facial expression feature and human pose feature. Our model includes a feature fusion method to incorporate these features with commonly used object features. The experiment results on public datasets demonstrate that our proposed model outperforms the baseline. Further experiments on paintings with abstract appearances and image corruptions show the promising performance of our proposed model.