First-Order Optimization Algorithms via Discretization of Finite-Time Convergent Flows

Mouhacine Benosman, Orlando Romero, Anoop Cherian

Keywords: Finite-time optimization, dynamical systems, deep neural networks optimization

Abstract: In this paper, we investigate the performance of several discretization algorithms for two first-order finite-time optimization flows. These flows are, namely, the rescaled-gradient flow (RGF) and the signed-gradient flow (SGF), and consist of non-Lipscthiz or discontinuous dynamical systems that converge locally in finite time to the minima of gradient-dominated functions. We introduce three discretization methods for these first-order finite-time flows, and provide convergence guarantees. We then apply the proposed algorithms in training neural networks and empirically test their performances on three standard datasets, namely, CIFAR10, SVHN, and MNIST. Our results show that our schemes demonstrate faster convergences against standard optimization alternatives, while achieving equivalent or better accuracy.

One-sentence Summary: Discretization of dynamical systems-based optimization algorithms with application to deep neural networks optimization.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=eV_81J3M5S