A Model Selection Framework for Learning Rate-Free Reinforcement Learning

ICML 2024 Workshop AutoRL Submission24 Authors

Keywords: Model Selection, Reinforcement Learning, Meta Learning

Abstract: The success of many reinforcement learning algorithms is dependent on the right choice of hyperparameters, with the learning rate being particularly influential. A suboptimal learning rate can hinder the algorithm's ability to converge. Mildly suboptimal choices may allow the algorithm to find an optimal policy only after requiring an extensive number of samples. In this work, we show the feasibility of using model selection meta-learning algorithms to select the best learning rates in reinforcement learning problems. We introduce the Model Selection Framework for Learning Rate-Free Reinforcement Learning and evaluate various model selection algorithms within our framework. Our results show that data-driven model selection strategies such as the D$^3$RB algorithm achieve better performance in the problem of learning rate selection for reinforcement learning algorithms, beating bandit strategies such as EXP3, and also standard hyperparameter selection methods such as the uniform sweep.

Submission Number: 24