Cost-aware learning of relevant contextual variables within Bayesian optimization

Julien Martinelli, Ayush Bharti, Sabina J. Sloman, Armi Tiihonen, Louis Filstroff, S. T. John, Samuel Kaski

Abstract:

Contextual Bayesian Optimization (CBO) is a powerful framework for optimizing
black-box, expensive-to-evaluate functions with respect to design variables, while
simultaneously efficiently integrating relevant contextual information regarding
the environment, such as experimental conditions. However, in many practical
scenarios, the relevance of contextual variables is not necessarily known beforehand.
Moreover, the contextual variables can sometimes be optimized themselves, a
setting that current CBO algorithms do not take into account. Optimizing contextual
variables may be costly, which raises the question of determining a minimal relevant
subset. In this paper, we frame this problem as a cost-aware model selection BO
task and address it using a novel method, Sensitivity-Analysis-Driven Contextual
BO (SADCBO). We learn the relevance of context variables by sensitivity analysis
of the posterior surrogate model at specific input points, whilst minimizing the
cost of optimization by leveraging recent developments on early stopping for BO.
We empirically evaluate our proposed SADCBO against alternatives on synthetic
experiments together with extensive ablation studies, and demonstrate a consistent
improvement across examples.