Convergence of adaptive algorithms for constrained weakly convex optimizationDownload PDF

21 May 2021, 20:51 (edited 26 Oct 2021)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: adaptive gradient algorithms, weakly convex optimization, AMSGrad, Adam
  • TL;DR: We establish convergence of adaptive algorithms for a class of nonsmooth nonconvex problems, for the first time.
  • Abstract: We analyze the adaptive first order algorithm AMSGrad, for solving a constrained stochastic optimization problem with a weakly convex objective. We prove the $\mathcal{\tilde O}(t^{-1/2})$ rate of convergence for the squared norm of the gradient of Moreau envelope, which is the standard stationarity measure for this class of problems. It matches the known rates that adaptive algorithms enjoy for the specific case of unconstrained smooth nonconvex stochastic optimization. Our analysis works with mini-batch size of $1$, constant first and second order moment parameters, and possibly unbounded optimization domains. Finally, we illustrate the applications and extensions of our results to specific problems and algorithms.
  • Supplementary Material: zip
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
8 Replies

Loading