Human action recognition by fusing deep features with Globality Locality Preserving Canonical Correlation Analysis

Nour El-Din El-Madany, Yifeng He, Ling Guan

Abstract: This paper proposes a novel Globality Locality Preserving Canonical Correlation Analysis (GLPCCA) for multiview learning. The proposed GLPCCA can preserve the global and local structures. Furthermore, we present a human action recognition framework by using GLPCCA to fuse depth and RGB modalities, which include the proposed Hierarchical Pyramid of Depth Motion Map Deep Convolutional Neural Network (HP-DMM-CNN) for the depth images, and Optical flow CNN for the RGB videos. The proposed framework was evaluated using two datasets, UTD Multimodal Human Action Dataset (UTD-MHAD) and SBU Kinect Interaction data set. The experimental results demonstrated that the proposed GLPCCA can achieve a higher average accuracy compared to several existing methods.