UQ-Guided Hyperparameter Optimization for Iterative Learners

Jiesong Liu, Feng Zhang, Jiawei Guan, Xipeng Shen

Keywords: Uncertainty quantification; Hyperparameter Optimization; iterative learners

Abstract:

Hyperparameter Optimization (HPO) plays a pivotal role in unleashing the potential of iterative machine learning models. This paper addresses a crucial aspect that has largely been overlooked in HPO: the impact of uncertainty in ML model training. The paper introduces the concept of uncertainty-aware HPO and presents a novel approach called the UQ-guided scheme for quantifying uncertainty. This scheme offers a principled and versatile method to empower HPO techniques in handling model uncertainty during their exploration of the candidate space.
By constructing a probabilistic model and implementing probability-driven candidate selection and budget allocation, this approach enhances the quality of the resulting model hyperparameters. It achieves a notable performance improvement of over 50\% in terms of accuracy regret and exploration time.

Primary Area: Optimization for deep networks

Submission Number: 4023