On the Marginal Regret Bound Minimization of Adaptive Methods

Wenjie Li, Guang Cheng

Keywords: Optimization Algorithm, Adaptive algorithms, Online Learning, Regret Minimization

Abstract: Numerous adaptive algorithms such as AMSGrad and Radam have been proposed and applied to deep learning recently. However, these modifications do not improve the convergence rate of adaptive algorithms and whether a better algorithm exists still remains an open question. In this work, we propose a new motivation for designing the proximal function of adaptive algorithms, named as marginal regret bound minimization. Based on such an idea, we propose a new class of adaptive algorithms that not only achieves marginal optimality but can also potentially converge much faster than any existing adaptive algorithms in the long term. We show the superiority of the new class of adaptive algorithms both theoretically and empirically using experiments in deep learning.

One-sentence Summary: We find a new motivation for designing adaptive algorithms that can be potentially much faster than existing algorithms

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=wlsb_rk1i1