Alignment of Visual Senses and Acoustical Senses based on Emotion Recognitions

Ja-Hwung Su, Yi-Wen Liao, Ming-Hung Kao, Yung-Wen Tsai, Chih-Jui Chang, Hsiu-Wei Wu, Cheng-Wei Chen

Abstract: Recently, due to the rapid advances on multimedia technology, multimedia data such as pictures and music has heavily invaded our daily life. By the mobile camera, users capture the interested scenes as the preferred pictures, and then browse these pictures in either computers or mobile devices. This incurs an interesting issue that, how to provide the users with a better browsing/listening atmosphere and experience by music. Actually, it can be viewed as the problem for aligning photos with music and also as an exciting research topic for cross-media retrieval. However, very few studies focus their attention on this topic. To aim at this issue, in this paper, we propose a novel cross-media alignment method that bridges visual pictures to harmonic music by visual emotions and acoustical emotions. In this method, the pictures and music are projected onto the emotion space first. Next, the emotion features are fuzzed to be near the human senses. Finally, the effective alignment method for associating pictures with music is performed. The results of subjective and objective evaluations on the real datasets reveal that, our proposed method can successfully provide the users with a rich experience on audiovisual presentations. Also, the proposed alignment algorithm is shown to be effective in terms of Normalized Discounted Cumulative Gain.