Understanding Deflation Process in Over-parametrized Tensor DecompositionDownload PDF

21 May 2021, 20:50 (edited 24 Oct 2021)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: tensor decomposition, overparametrization, gradient flow, implicit regularization
  • TL;DR: The paper analyzes gradient flow on overparametrized tensor decomposition and proves that its dynamics is similar to a tensor deflation process.
  • Abstract: In this paper we study the training dynamics for gradient flow on over-parametrized tensor decomposition problems. Empirically, such training process often first fits larger components and then discovers smaller components, which is similar to a tensor deflation process that is commonly used in tensor decomposition algorithms. We prove that for orthogonally decomposable tensor, a slightly modified version of gradient flow would follow a tensor deflation process and recover all the tensor components. Our proof suggests that for orthogonal tensors, gradient flow dynamics works similarly as greedy low-rank learning in the matrix setting, which is a first step towards understanding the implicit regularization effect of over-parametrized models for low-rank tensors.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
  • Code: zip
14 Replies