The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning

Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, Adam M White

Keywords: Reinforcement Learning, benchmark

TL;DR: Develops a new benchmark for reinforcement learning, allowing statistically significant and meaningful claims while being computationally efficient. This benchmark motivates algorithm development which is robust across environments.

Abstract: This paper introduces a new benchmark, the Single Hyperparameter Benchmark, that allows comparison of RL algorithms across environments using only a single hyperparameter, encouraging algorithmic development which is insensitive to hyperparameters. We demonstrate that the benchmark is robust to statistical noise and obtains qualitatively similar results across repeated applications, even when using a small number of samples. This robustness makes the benchmark computationally cheap to apply, allowing statistically sound insights at a low cost. To demonstrate the applicability of the SHB to modern RL algorithms on challenging environments, we provide a novel empirical study of an open question in the continuous control literature. We show, with high confidence, that there is no meaningful difference in performance between Ornstein-Uhlenbeck noise and uncorrelated Gaussian noise for exploration with the DDPG algorithm across the entire DMControl suite.