Efficient Model Performance Estimation via Feature Histories

Shengcao Cao, Xiaofang Wang, Kris M. Kitani

Keywords: Hyperparameter Optimization, Neural Architecture Search

Abstract: An essential step in the task of model selection, such as hyper-parameter optimization (HPO) or neural architecture search (NAS), is the process of estimating a candidate model's (hyper-parameter or architecture) performance. Due to the high computational cost of training models until full convergence, it is necessary to develop efficient methods that can accurately estimate a model's best performance using only a small time budget. To this end, we propose a novel performance estimation method which uses a history of model features observed during the early stages of training to obtain an estimate of final performance. Our method is versatile. It can be combined with different search algorithms and applied to various configuration spaces in HPO and NAS. Using a sampling-based search algorithm and parallel computing, our method can find an architecture which is better than DARTS and with an 80\% reduction in search time.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We propose a performance estimation strategy using feature histories which improves search algorithms in hyperparameter optimization and neural architecture search tasks.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=UIrOfaDD2w