PriorBand: HyperBand + Human Expert Knowledge

Neeratyoy Mallik, Carl Hvarfner, Danny Stoll, Maciej Janowski, Eddie Bergman, Marius Lindauer, Luigi Nardi, Frank Hutter

Keywords: Hyperparameter Optimization, Low Budget, Cheap Proxy Tasks, Expert Beliefs

TL;DR: To enable Hyperparameter Optimization for expensive Deep Learning pipelines, we propose an algorithm that can leverage cheap proxy tasks and expert priors.

Abstract:

Hyperparameters of Deep Learning (DL) pipelines are crucial for their performance. While a large number of methods for hyperparameter optimization (HPO) have been developed, they are misaligned with the desiderata of a modern DL researcher. Since often only a few trials are possible in the development of new DL methods, manual experimentation is still the most prevalent approach to set hyperparameters,
relying on the researcher’s intuition and cheap preliminary explorations. To resolve this shortcoming of HPO for DL, we propose PriorBand, an HPO algorithm tailored to DL, able to utilize both expert beliefs and cheap proxy tasks. Empirically, we demonstrate the efficiency of PriorBand across a range of DL models and tasks using as little as the cost of 10 training runs and show its robustness against poor expert beliefs and misleading proxy tasks.