Sparse low-rank fusion based deep features for missing modality face recognition

Ming Shao, Zhengming Ding, Yun Fu

Abstract: Multi-modality data recently attract more and more research attention. In this paper, we concentrate on a very interesting problem - image classification with missing modality. Specifically, only images in one modality as well as a relevant auxiliary database are accessible during the training phase, which is significantly different from general image classification under the same modality. To this end, we propose a novel framework integrating multiple deep autoencoders with bagging strategy. For each autoencoder, we generate its input by randomly sampling data from other modality and the auxiliary database, and enforce its output to lie in a common feature space through Robust PCA. Finally, a novel sparse low-rank feature fusion approach is proposed in the test phase to integrate multiple features learned from different autoencoders, followed by a decision voting. Extensive experiments on two databases, i.e., BUAA-NIRVIS, Oulu-CASIA NIRVIS databases demonstrate the effectiveness of the proposed framework when there is only one modality available for training.