MSL: An Adaptive Momentem-based Stochastic Line-search Framework

Chen Fan, Sharan Vaswani, Christos Thrampoulidis, Mark Schmidt

Keywords: line-search, SGD with momentum, Adam, Armijo condition

Abstract:

Various adaptive step sizes have been proposed recently to reduce the amount of tedious manual
tuning. A popular example is back-tracking line-search based on a stochastic Armijo condition.
But the success of this strategy relies crucially on the search direction being a descent direction.
Importantly, this condition is violated by both SGD with momentum (SGDM) and Adam, which are
common choices in deep-net training. Adaptively choosing the step size in this setting is thus non-
trivial and less explored despite its practical relevance. In this work, we propose two frameworks,
namely, momentum correction and restart, that allow the use of stochastic line-search in conjunction
with a generalized Armijo condition, and apply them to both SGDM and Adam. We empirically
verify that the proposed algorithms are robust to the choice of the momentum parameter and other
hyperparameters.

Submission Number: 38