Accelerating First-Order Optimization Algorithms

Ange Tato, Roger Nkambou

TL;DR: Accelerating First-Order Optimization Algorithms

Abstract: Several stochastic optimization algorithms are currently available. In most cases, selecting the best optimizer for a given problem is not an easy task. Therefore, instead of looking for yet another ’absolute’ best optimizer, accelerating existing ones according to the context might prove more effective. This paper presents a simple and intuitive technique to accelerate first-order optimization algorithms. When applied to first-order optimization algorithms, it converges much more quickly and achieves lower function/loss values when compared to traditional algorithms. The proposed solution modifies the update rule, based on the variation of the direction of the gradient during training. Several tests were conducted with SGD, AdaGrad, Adam and AMSGrad on three public datasets. Results clearly show that the proposed technique, has the potential to improve the performance of existing optimization algorithms.

Code: https://github.com/angetato/Custom-Optimizer-on-Keras

Keywords: Neural Networks, Gradient Descent, First order optimization