Efficient First Order Method for Saddle Point Problems with Higher Order Smoothness

Nuozhou Wang, Junyu Zhang, Shuzhong Zhang

Abstract: This paper studies the complexity of finding approximate stationary points for the smooth nonconvex-strongly-concave (NC-SC) saddle point problem: \min_x\max_yf(x,y). Under the standard first-order smoothness conditions where f is \ell-smooth in both arguments and \mu_y-strongly concave in y, existing literature shows that the optimal complexity for first-order methods to obtain an \epsilon-stationary point is \tilde{O}\big(\sqrt{\kappa_y}\ell\epsilon^{-2}\big), where \kappa_y=\ell/\mu_y is the condition number. However, when \Phi(x):=\max_y f(x,y) has L_2-Lipschitz continuous Hessian in addition, we derive a first-order algorithm with an \tilde{O}\big(\sqrt{\kappa_y}\ell^{1/2}L_2^{1/4}\epsilon^{-7/4}\big) complexity by designing an accelerated proximal point algorithm enhanced with the "Convex Until Proven Guilty" technique. Moreover, an improved \Omega\big(\sqrt{\kappa_y}\ell^{3/7}L_2^{2/7}\epsilon^{-12/7}\big) lower bound for first-order method is also derived for sufficiently small \epsilon. As a result, given the second-order smoothness of the problem, the complexity of our method improves the state-of-the-art result by a factor of \tilde{O}\big(\big(\frac{\ell^2}{L_2\epsilon}\big)^{1/4}\big), while almost matching the lower bound except for a small \tilde{O}\big(\big(\frac{\ell^2}{L_2\epsilon}\big)^{1/28}\big) factor.