Implicit Regularization in Deep Tucker Factorization: Low-Rankness via Structured Sparsity

Kais Hariz, Hachem Kadri, Stéphane Ayache, Maher Moakher, Thierry Artières

Abstract: We theoretically analyze the implicit regularization of deep learning for tensor completion. We show that deep Tucker factorization trained by gradient descent induces a structured sparse regularization. This leads to a characterization of the effect of the depth of the neural network on the implicit regularization and provides a potential explanation for the bias of gradient descent towards solutions with low multilinear rank. Numerical experiments confirm our theoretical findings and give insights into the behavior of gradient descent in deep tensor factorization.