Re-examining Supervised Dimension Reduction for High-Dimensional Bayesian Optimization

Quanlin Chen, Jing Huo, Yiyu Chen, Tianyu Ding, Yang Gao, Dong Li, Xu He

Abstract: Bayesian optimization (BO) has been broadly applied to optimize expensive-to-evaluate black-box functions, but it is still challenging to scale BO to high dimensions while retaining sample efficiency. A solution in the existing literature is to assume that there exists a lower-dimensional structure for objective functions and learn the lower-dimensional embedding via supervised dimension reduction. For example, BO based on Sliced Inverse Regression (SIR) directly uses SIR to discover the intrinsic lower-dimensional structure of the objective function. However, the assumption of SIR leads to a mismatch in BO, and maximizing a high-dimensional acquisition function also leads to its poor performance. To reduce the mismatch between dimension reduction methods and BO, we introduce Kernel Dimension Reduction (KDR) and manifold KDR to BO. Furthermore, to improve the performance of acquisition functions, we construct a constrained low-dimensional acquisition function, where the constraint is constructed by the inverse mapping from the central subspace back to the original space using a batch of Gaussian Process models. We verify empirically that tackling these two issues improves the performance of methods based on supervised dimension reduction on a wide range of problems.