A Geometric Structure of Acceleration and Its Role in Making Gradients Small FastDownload PDF

Published: 09 Nov 2021, Last Modified: 05 May 2023NeurIPS 2021 PosterReaders: Everyone
Keywords: acceleration, convex optimization, Euclidean geometry, gradient norm, small gradients, making gradients small, composite optimization, OGM, FISTA, OGM-G, potential function-based, Lyapunov analysis, complexity bounds
TL;DR: We find a geometric structure of acceleration and use it to obtain a method for making gradients small at rate $\mathcal{O}(1/K^4)$ in the prox-grad setup.
Abstract: Since Nesterov's seminal 1983 work, many accelerated first-order optimization methods have been proposed, but their analyses lacks a common unifying structure. In this work, we identify a geometric structure satisfied by a wide range of first-order accelerated methods. Using this geometric insight, we present several novel generalizations of accelerated methods. Most interesting among them is a method that reduces the squared gradient norm with $\mathcal{O}(1/K^4)$ rate in the prox-grad setup, faster than the $\mathcal{O}(1/K^3)$ rates of Nesterov's FGM or Kim and Fessler's FPGM-m.
Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
Supplementary Material: pdf
9 Replies