Quasi-Newton Methods for Saddle Point ProblemsDownload PDF

Published: 31 Oct 2022, 18:00, Last Modified: 11 Jan 2023, 11:58NeurIPS 2022 AcceptReaders: Everyone
Keywords: Minimax Optimization, Quasi-Newton
Abstract: This paper studies quasi-Newton methods for strongly-convex-strongly-concave saddle point problems. We propose random Broyden family updates, which have explicit local superlinear convergence rate of ${\mathcal O}\big(\big(1-1/(d\varkappa^2)\big)^{k(k-1)/2}\big)$, where $d$ is the dimension of the problem, $\varkappa$ is the condition number and $k$ is the number of iterations. The design and analysis of proposed algorithm are based on estimating the square of indefinite Hessian matrix, which is different from classical quasi-Newton methods in convex optimization. We also present two specific Broyden family algorithms with BFGS-type and SR1-type updates, which enjoy the faster local convergence rate of $\mathcal O\big(\big(1-1/d\big)^{k(k-1)/2}\big)$. Our numerical experiments show proposed algorithms outperform classical first-order methods.
Supplementary Material: pdf
13 Replies