Optimal Designs of Gaussian Processes with Budgets for Hyperparameter Optimization

Yimin Huang, Yujun Li, Zhenguo Li, Zhihua Zhang

Keywords: Automated Hyperparameter Optimization, Budgets, Efficiency, Optimal Initial Design, Robustness

Abstract: The remarkable performance of modern deep learning methods depends critically on the optimization of their hyperparameters. One major challenge is that evaluating a single hyperparameter configuration on large datasets could nowadays easily exceed hours or days. For efficient sampling and fast evaluation, some previous works presented effective computing resource allocation schemes and built a Bayesian surrogate model to sample candidate hyperparameters. However, the model itself is not related to budgets which are set manually. To deal with this problem, a new Gaussian Process model involved in budgets is proposed. Further, for this model, an optimal design is constructed by the equivalence theorem to replace random search as an initial sampling strategy in the search space. Experiments demonstrate that the new model has the best performance among competing methods. Moreover, comparisons between different initial designs with the same model show the advantage of the proposed optimal design.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: This paper proposes a new Guassian Process model involved with budgets and matched optimal initial designs to improving the efficiency and robustness for hyperparameter optimization.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=2UVV5KIrTL