Randomized Subspace Methods for Optimization on Fixed-Rank Matrix Manifolds

David Huckleberry Gutman, Nam Ho-Nguyen

Keywords: Fixed-rank, Riemannian manifold, randomized, subspace

TL;DR: New randomized methods with low storage and iteration costs for Riemannian optimization over the manifolds of fixed-rank matrices and fixed-rank positive semi-definite matrices.

Abstract: This paper provides the first randomized subspace methods for optimization over fixed-rank matrix manifolds. This allows us to avoid expensive full matrix decompositions to ensure efficient exponential map computations via at most a $2\times 2$ eigendecomposition and rank-one updates, with low storage costs. To facilitate this, we analyze the geometries of fixed-rank matrix manifolds as Riemannian quotients of convenient product manifolds. Due to the quotient structure, subspaces of interest correspond exactly to those in the horizontal space of said product manifolds. A tangent subspace descent scheme is then devised by decomposing the horizontal space into orthogonal subspaces. Existing instances of tangent subspace descent depend upon the selection of a subspace from a fixed collection of ones that vary smoothly over the entire manifold. In sharp contrast to these instances on other manifolds, subspaces in our scheme are not selected from any such smoothly varying collection. Instead, the randomly selected subspace at the current iterate is carefully constructed based on the past iterates and their accompanying subspace selections. Experiments for the trace regression problem demonstrate the superiority of the methods relative to full gradient methods in terms of both CPU time and iterate count.

Primary Area: optimization

Submission Number: 8136