Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Improving Stochastic Gradient Descent with Feedback
Jayanth Koushik, Hiroaki Hayashi
Nov 04, 2016 (modified: Nov 16, 2016)ICLR 2017 conference submissionreaders: everyone
Abstract:In this paper we propose a simple and efficient method for improving stochastic gradient descent methods by using feedback from the objective function. The method tracks the relative changes in the objective function with a running average, and uses it to adaptively tune the learning rate in stochastic gradient descent. We specifically apply this idea to modify Adam, a popular algorithm for training deep neural networks. We conduct experiments to compare the resulting algorithm, which we call Eve, with state of the art methods used for training deep learning models. We train CNNs for image classification, and RNNs for language modeling and question answering. Our experiments show that Eve outperforms all other algorithms on these benchmark tasks. We also analyze the behavior of the feedback mechanism during the training process.
TL;DR:We improve stochastic gradient descent by incorporating feedback from the objective function
Keywords:Deep learning, Optimization
Enter your feedback below and we'll get back to you as soon as possible.