A new accelerated gradient method inspired by continuous-time perspective

Yasong Feng, Weiguo Gao

Keywords: accelerated gradient method, matrix completion, first-order methods, differential equation

Abstract: Nesterov's accelerated method are widely used in problems with machine learning background including deep learning. To give more insight about the acceleration phenomenon, an ordinary differential equation was obtained from Nesterov's accelerated method by taking step sizes approaching zero, and the relationship between Nesterov's method and the differential equation is still of research interest. In this work, we give the precise order of the iterations of Nesterov's accelerated method converging to the solution of derived differential equation as step sizes go to zero. We then present a new accelerated method with higher order. The new method is more stable than ordinary method for large step size and converges faster. We further apply the new method to matrix completion problem and show its better performance through numerical experiments.

One-sentence Summary: We give the precise order of the iterations of Nesterov's accelerated method converging to the solution of derived differential equation and present a new method with application in matrix completion.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=wjsy_TadIS