Mechanic: A Learning Rate Tuner

Ashok Cutkosky, Aaron Defazio, Harsh Mehta

Keywords: optimization, deep learning, online convex optimization

TL;DR: A meta-algorithm that tunes the LR of any base optimizer, with empirical evaluation.

Abstract: We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call Mechanic. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate Mechanic on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, Mechanic either comes very close to, matches or even improves upon manual tuning of learning rates.

Submission Number: 9112