On the Optimality Gap of Warm-Started Hyperparameter Optimization

Parikshit Ram

Abstract: We study the general framework of warm-started hyperparameter optimization (HPO) where we have some source datasets (tasks) where we have already performed HPO, and we wish to leverage the results of these HPO to warm-start the HPO on an unseen target dataset (and perform few-shot HPO). Various meta-learning schemes have been proposed over the last decade (and more) for this problem. In this paper, we theoretically analyse the optimality gap of the hyperparameter obtained via such warm-started few-shot HPO, and provide novel results for multiple existing meta-learning schemes. We show how these results allow us identify situations where certain schemes have advantage over others.

Keywords: Meta learning, warm-starting, hyperparameter optimization, theoretical analysis, optimality gap

One-sentence Summary: We provide theoretical guarantees of various meta-learning based warm-started hyperparameter optimization

Track: Main track

Reproducibility Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Reviewers: Already reviewer

CPU Hours: 0

GPU Hours: 0

TPU Hours: 0

Evaluation Metrics: No

Class Of Approaches: Meta-Learning, Sequential Model Based Optimization, Optimality Gap Analysis

Datasets And Benchmarks: NA

Performance Metrics: NA

Steps For Environmental Footprint Reduction During Development: NA

Estimated CO2e Footprint: 0

Benchmark Performance: NA

Benchmark Time: NA