Combination of Pyramid CNN Representation and Spatial-Temporal Representation for Facial Expression Recognition

Shulin Xu, Nan Pu, Li Qian, Guoqiang Xiao

Abstract: In this paper, we propose a novel framework for facial expression recognition in video sequences by combining deep convolutional feature and spatial-temporal feature. Firstly, apex frame of every sequence is selected adaptively by calculating the displacement of facial landmarks. Then, pyramid CNN-based feature is extracted on the apex frame to capture the information of global and local regions of human face. Afterwards, spatial-temporal LBP-TOP feature is generated from video sequence and integrated with pyramid CNN-based feature to represent video, which reflect dynamic and static texture information of facial expressions. Finally, the multiclass support vector machine (SVM) with one-versus-one strategy is applied to classify facial expressions. Experimental results on the extended Cohn-Kanade (CK+) and Oulu-CASIA datasets demonstrate the superiority of our proposed method.