Cost-Efficient Training for Automated Algorithm Selection

Erdem Kuş, Nguyen Dang, Ozgur Akgun, Ian Miguel

Keywords: Automated Algorithm Selection, Active Learning, Constraint Programming

TL;DR: This research proposes an active learning approach that attempts to reduce the labelling cost by using only a subset of the training data with timeout predictor and dynamic timeout configurations.

Abstract: When solving decision and optimisation problems, many competing algorithms have complementary strengths. Typically, there is no single algorithm that works well for all instances of a problem. Automated algorithm selection has been shown to work very well for choosing a suitable algorithm for a given instance. However, the cost of training can be prohibitively large due to the need of running all candidate algorithms on a set of training instances. In this work, we explore reducing this cost by selecting specific instance/algorithm combinations to train on, rather than requiring all algorithms for all instances. We approach this problem in three ways: using active learning to decide based on prediction uncertainty, augmenting the algorithm predictors with a timeout predictor, and collecting training data using a progressively increasing timeout. We evaluate combinations of these approaches on six datasets from ASLib and present the reduction in labelling cost.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.

Evaluation Metrics: No

Submission Number: 24