Importance of tuning hyperparameters of machine learning algorithms

Andreas C Mueller, Hilde Jacoba Petronella Weerts, Joaquin Vanschoren

Abstract:

The performance of many machine learning algorithms depends
on their hyperparameter settings. The goal of this study is to determine whether it is important to tune a hyperparameter or whether
it can be safely set to a default value. We present a methodology
to determine the importance of tuning a hyperparameter based on
a non-inferiority test and tuning risk: the performance loss that is
incurred when a hyperparameter is not tuned, but set to a default
value. Because our methods require the notion of a default parameter, we present a simple procedure that can be used to determine
reasonable default parameters. We apply our methods in a benchmark study using 59 datasets from OpenML. Our results show that
leaving particular hyperparameters at their default value is noninferior to tuning these hyperparameters. In some cases, leaving
the hyperparameter at its default value even outperforms tuning it
using a search procedure with a limited number of iterations.