Escaping saddle points efficiently in equality-constrained optimization problems

Yue Sun, Maryam Fazel

Abstract:

We consider minimizing a nonconvex, smooth
function f(x) subject to equality constraints
ci(x) = 0 (equivalently, x ∈ M where M is
a smooth manifold). We show that a perturbed
version of the gradient projection algorithm con-
verges to a second-order stationary point for this
problem (and hence is able to escape saddle points
on the manifold) in a number of iterations that de-
pend only polylogarithmically on the dimension
(hence is almost dimension-free). This matches
a rate known only for unconstrained smooth min-
imization. While the unconstrained case is well-
studied, our result is the first to prove such a
rate for a constrained problem, which includes
examples such as PCA, Burer-Monteiro factor-
ized SDPs, and more. The rate of convergence
depends as 1/2 on the accuracy , and also de-
pends polynomially on appropriate smoothness
and curvature parameters for the cost function and
the constraints – we define these parameters using
the explicit form of the constraints ci(x) = 0 (in a
representation-dependent fashion), but also briefly
examine a geometric alternative for the manifold
curvature. Future work will examine this geo-
metric setting further, and will consider the more
challenging problems of inequality constraints.