Rethinking of Encoder-based Warm-start Methods in Hyperparameter Optimization

Dawid Płudowski, Antoni Zajko, Katarzyna Woźnica, Anna Kozak

Keywords: meta-learning, encoders, hyperparameter optimization, dataset representations, meta-features

TL;DR: This research explores using dataset encoders to learn representations for warm-starting of the Bayesian Hyperparameter Optimization on tabular data, finding that general representations may not be suitable for this task.

Abstract:

Effectively representing heterogeneous tabular datasets for meta-learning purposes remains an open problem. Previous approaches rely on predefined meta-features, for example, statistical measures or landmarkers. The emergence of dataset encoders opens new possibilities for the extraction of meta-features because they do not involve any handmade design. Moreover, they are proven to generate dataset representations with desired spatial properties. In this research, we evaluate an encoder-based approach to one of the most established meta-tasks - warm-starting of the Bayesian Hyperparameter Optimization. To broaden our analysis we introduce a new approach for representation learning on tabular data based on [Iwata and Kumagai, 2020]. The validation on over 100 datasets from UCI and an independent metaMIMIC set of datasets highlights the nuanced challenges in representation learning. We show that general representations may not suffice for some meta-tasks where requirements are not explicitly considered during extraction.

[Iwata and Kumagai, 2020] Tomoharu Iwata and Atsutoshi Kumagai. Meta-learning from Tasks with Heterogeneous Attribute Spaces. In Advances in Neural Information Processing Systems, 2020.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.

CPU Hours: 8

GPU Hours: 0

TPU Hours: 0

Submission Number: 18