An Attention-based Approach for Bayesian Optimization with Dependencies
Keywords: Bayesian optimization; Search space with dependencies; Attention; Deep kernel learning;
Abstract: Bayesian Optimization (BO) is a sample-efficient method for optimizing black-box problems that are expensive to evaluate. The canonical BO is conducted in search spaces where hyperparameters are independent and the dimension of configurations remains fixed. However, different algorithms typically require their distinct hyperparameters in practice, thereby yielding a hierarchical search space structure. Such a nested configuration challenges the direct application of Bayesian optimization, as it obscures the independence assumptions made in the standard Bayesian optimization formulation. In this paper, we propose a structure-aware embedding and an attention-based Deep Kernel Gaussian Process to capture the response surface in such conditional search spaces. By endowing the surrogate model with context on the conditional structure, our approach facilitates Bayesian optimization in navigating nested hyperparameter configurations. Empirical results on both a tree-structured simulation benchmark and several real-world benchmarks demonstrate that our proposed approach improves the efficacy and efficiency of BO in conditional search spaces.
Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3175
Loading