Rethink Autoencoders: Robust Manifold Learning

Taihui Li, Rishabh Mehta, Zecheng Qian, Ju Sun

Abstract:

PCA can be made robust to data corruption, i.e., robust PCA. What about the deep autoencoder, as a nonlinear generalization of PCA? This further motivates us to “reinvent” a factorization-based PCA as well as its nonlinear generalization. Focusing on sparse corruption, we model the sparsity structure explicitly using the L1 norm to obtain various robust formulations. For linear data, robust factorization performs comparably to the seminal convex formulation of robust PCA, whereas robust autoencoders provably fail. For nonlinear data, we perform a careful experimental evaluation of robust deep autoencoders and robust nonlinear factorization for corruption removal on natural
images. Both schemes can remove a considerable level of sparse corruption and effectively reconstruct the clean images.