Optimal and Adaptive Monteiro-Svaiter AccelerationDownload PDF

Published: 31 Oct 2022, Last Modified: 14 Jan 2023NeurIPS 2022 AcceptReaders: Everyone
Keywords: convex optimization, optimization theory, second-order methods, Monteiro-Svaiter acceleration, proximal points, momentum, Newton's method, cubic regularization, conjugate residuals, oracle complexity, optimal algorithms, adaptive methods, parameter-free methods
TL;DR: Monteiro-Svaiter acceleration, without the agonizing bisection (and the log term, and the parameter-tuning)
Abstract: We develop a variant of the Monteiro-Svaiter (MS) acceleration framework that removes the need to solve an expensive implicit equation at every iteration. Consequently, for any $p\ge 2$ we improve the complexity of convex optimization with Lipschitz $p$th derivative by a logarithmic factor, matching a lower bound. We also introduce an MS subproblem solver that requires no knowledge of problem parameters, and implement it as either a second- or first-order method by solving linear systems or applying MinRes, respectively. On logistic regression problems our method outperforms previous accelerated second-order methods, but under-performs Newton's method; simply iterating our first-order adaptive subproblem solver is competitive with L-BFGS.
Supplementary Material: pdf
12 Replies