Robust and fast low-rank deep convolutional feature recovery: toward information retention and accelerated convergence

Jiahuan Ren, Zhao Zhang, Jicong Fan, Haijun Zhang, Mingliang Xu, Meng Wang

Abstract: Notwithstanding the great progress on deep convolutional neural networks (CNNs) has been made during last decade, the representation ability may still be restricted and it usually needs more epochs to converge in training, due to the information loss caused by the up-/down-sampling operations. In this paper, we propose a general deep feature recovery layer, termed Low-rank Deep Feature Recovery (LDFR), to enhance the representation of convolutional features by seamlessly integrating the low-rank recovery into CNNs, which can be easily extended to all CNNs-based models. To be specific, to recover the lost useful information, LDFR learns the low-rank projections to embed feature maps onto a low-rank subspace based on the selected informative convolutional feature maps. Such operation can ensure all the convolutional feature maps to be reconstructed easily to recover the underlying subspace, with more useful detailed information discovered, e.g., the strokes of characters or the texture information of clothes. To make the learnt low-rank subspaces more powerful for feature recovery, we design a fusion strategy to obtain a generalized subspace, which averages over all learnt subspaces in each LDFR layer, so that the convolutional features in test phase can be recovered effectively via low-rank embedding. We also present a fast version of LDFR, called FLDFR, to speedup the optimization of LDFR by flattening all feature maps of batch images to recover the lost information. Extensive simulations on several image datasets show that the existing CNN models equipped with our LDFR layers can obtain better performance.