Getting the Best Bang For Your Buck: Choosing What to Evaluate for Faster Bayesian Optimization

Md Shahriar Iqbal, Jianhai Su, Lars Kotthoff, Pooyan Jamshidi

Abstract: Machine learning system design frequently necessitates balancing multiple objectives, such as prediction error and energy consumption for deep neural networks (DNNs). Typically, no single design performs well across all objectives; thus, finding Pareto-optimal designs is of interest. Measuring different objectives frequently incurs different costs; for example, measuring the prediction error of DNNs is significantly more expensive than measuring the energy consumption of a pre-trained DNN because it requires re-training the DNN. Current state-of-the-art methods do not account for this difference in objective evaluation cost, potentially wasting costly evaluations of objective functions for little information gain. To address this issue, we propose a novel cost-aware decoupled approach that weights the improvement of the hypervolume of the Pareto region by the measurement cost of each objective. We perform experiments on a of range of DNN applications for comprehensive evaluation of our approach.

Keywords: Multi-objective Optimization, Decoupled Optimization, Cost-Awareness, DNN system stack

One-sentence Summary: A novel cost-aware decoupled multi-objective Bayesian optimization approach that takes into account the non-uniformity of objective evaluation costs.

Reproducibility Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Reviewers: Md Shahriar Iqbal, [e-mail]

Code And Dataset Supplement: https://github.com/anonpassen/anonoptimization