Safely Exploring Large Momentum Steps with Stochastic Curve Searches

TMLR Paper8654 Authors

Abstract:

The use of stochastic line searches has emerged as an effective safeguard strategy for employing large learning rates in the training of deep models via stochastic gradient descent. However, exploiting this approach with different search directions is not straightforward; momentum type directions, in particular, pose several challenges in this regard both from the theoretical and the computational sides.
In this work, we present stochastic curve search (SCS) as a generalization of the stochastic line search. SCS allows to evaluate updates along directions that may not be of descent, while still ensuring the sufficient decrease of the mini-batch objective at each iteration.
We show that the proposed framework is well-defined and that, under standard assumptions, the method converges in expectation.
We also empirically establish that using SCS alongside several momentum based algorithms allows the employment of aggressive hyperparameters, improving either the stability or the speed of the training process. The resulting algorithmic framework is demonstrated to perform competitively against state-of-the-art methods, achieving interesting results in terms of both efficiency and effectiveness across a diverse set of learning benchmarks.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Nicolas_Loizou1

Submission Number: 8654