Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
FX OPTIMIZER: SUCCESSOR TO ADAM OPTIMIZER USING BATCH WISE MOVING AVERAGES.
Hemen Ashodia, Tanisha R Bhayani
Feb 12, 2018 (modified: Feb 12, 2018)ICLR 2018 Workshop Submissionreaders: everyone
Abstract:The random distribution of the input data creates an imbalance in learning parameters, this causes some inputs’ patterns to be learned fast while others to be slow, and because of this, there is a subtle increase in the time the parameters takes to
converge, this could be improved by storing the moving averages of the individual batches of the trainable variables. This lets the parameter update step to use the previous information about the same batch to update the parameters in the current batch. The paper presents an optimizer named Fx Optimizer, which is a first order optimizer, similar to ADAM Optimizer in the property but maintains the batch moving averages for each weight and converges faster and better than ADAM. This is shown experimentally on MNIST data-set. The intuition of the Optimizer comes from how the brain may take more or less time to earn different types of inputs from same class based on high or low complexity of input.
TL;DR:Fx Optimizer, which is a first order optimizer, similar to ADAM Optimizer in the property but maintains the batch moving averages for each weight and converges faster and better than ADAM.
Keywords:Neural Network Optimization, Moving Averages, Adaptive Stochastic Optimization, First Order Optimizers, ADAM Optimizer
Enter your feedback below and we'll get back to you as soon as possible.