Guarantees for Alternating Least Squares in Overparameterized Tensor Decompositions

Dionysis Arvanitakis, Vaidehi Srinivas, Aravindan Vijayaraghavan

Keywords: tensor decomposition, overparameterization, alternating least squares, optimization, iterative methods

TL;DR: We prove that ALS with overparameterized rank $k=r^2$ achieves global convergence for decomposing rank $r$ tensors

Abstract: Tensor decomposition is a canonical non-convex optimization problem that is computationally challenging, and yet important due to applications in factor analysis and parameter estimation of latent variable models. In practice, scalable iterative methods, particularly Alternating Least Squares (ALS), remain the workhorse for tensor decomposition despite the lack of global convergence guarantees. A popular approach to tackle challenging non-convex optimization problems is overparameterization--- on input an $n \times n \times n$ tensor of rank $r$, the algorithm can output a decomposition of potentially rank $k$ (potentially larger than $r$). On the theoretical side, overparameterization for iterative methods is challenging to reason about and requires new techniques. The work of Wang et al., (NeurIPS 2020) makes progress by showing that a variant of gradient descent globally converges when overparameterized to $k=O(r^{7.5} \log n)$. Our main result shows that overparameterization provably enables global convergence of ALS: on input a third order $n \times n \times n$ tensor with a decomposition of rank $r \ll n$, ALS overparameterized with rank $k=O(r^2)$ achieves global convergence with high probability under random initialization. Moreover our analysis also gives guarantees for the more general low-rank approximation problem. The analysis introduces new techniques for understanding iterative methods in the overparameterized regime based on new matrix anticoncentration arguments.

Primary Area: Optimization (e.g., convex and non-convex, stochastic, robust)

Submission Number: 24014