Revisiting Learning Rate Control

Micha Henheik, Theresa Eimer, Marius Lindauer

Confirmation: our paper adheres to reproducibility best practices. In particular, we confirm that all important details required to reproduce results are described in the paper,, the authors agree to the paper being made available online through OpenReview under a CC-BY 4.0 license (https://creativecommons.org/licenses/by/4.0/), and, the authors have read and commit to adhering to the AutoML 2025 Code of Conduct (https://2025.automl.cc/code-of-conduct/).

TL;DR: We test different learning rate control paradigms (MF HPO, schedules, tuning-free methods) and find all of them are brittle and situational, but selecting between them would yield great and efficient performances.

Abstract:

The learning rate is one of the most important hyperparameters in deep learning, and how to control it is an active area within both AutoML and deep learning research.
Approaches for learning rate control span from classic optimization to online scheduling based on gradient statistics.
This paper compares paradigms to assess the current state of learning rate control.
We find that methods from multi-fidelity hyperparameter optimization, fixed-hyperparameter schedules, and hyperparameter-free learning often perform very well on selected deep learning tasks but are not reliable across settings.
This highlights the need for algorithm selection methods in learning rate control, which have been neglected so far by both the AutoML and deep learning communities.
We also observe a trend of hyperparameter optimization approaches becoming less effective as models and tasks grow in complexity, even when combined with multi-fidelity approaches for more expensive model trainings.
A focus on more relevant test tasks and new promising directions like finetunable methods and meta-learning will enable the AutoML community to significantly strengthen its impact on this crucial factor in deep learning.

Keywords: deep learning, learning rates, hyperparamenter-free optimization

Submission Number: 2