A Practical Descent Method for Singular Value Decomposition

Longhao Wang, Mingzhen He, Ruikai Yang, Xiaolin Huang

Keywords: Descent Methods, Singular Value Decomposition (SVD), Optimization problem, Primal-dual relationship

Abstract:

Singular Value Decomposition (SVD) is a long-established technique, with most existing methods relying on matrix-based formulations. However, matrix operations are relatively less friendly to parallelization and distributed computation compared to descent-based methods, motivating the need for alternative approaches. Descent-based methods offer a promising direction, yet existing ones such as Riemannian gradient descent suffer from inefficiency due to the need for repeated projections onto nonlinear manifolds.
In this work, we introduce a novel descent method for SVD grounded in a primal–dual reformulation. Specifically, we construct a least-squares primal problem whose dual corresponds to the SVD. We show that (i) the non-zero KKT solutions of the primal problem yield the singular vectors of the matrix, and (ii) inexact singular value estimation still ensures bounded reconstruction error. Building on these results, we propose an iterative descent-based algorithm, Des-SVD, along with scalable variants leveraging random sampling and parallelization.
Extensive experiments demonstrate that Des-SVD achieves significantly higher computational efficiency compared to prior descent methods, while remaining competitive with matrix-based algorithms. Our implementation is publicly available at https://anonymous.4open.science/r/Descent-SVD-method.

Primary Area: optimization

Submission Number: 7072