High-Dimensional Bayesian Optimization via Tree-Structured Additive Models

Abstract

Bayesian Optimization (BO) has shown significant success in tackling expensive low-dimensional black-box optimization problems. Many optimization problems of interest are high-dimensional, and scaling BO to such settings remains an important challenge. In this paper, we consider generalized additive models in which low-dimensional functions with overlapping subsets of variables are composed to model a high-dimensional target function. Our goal is to lower the computational resources required and facilitate faster model learning by reducing the model complexity while retaining the sample-efficiency of existing methods. Specifically, we constrain the underlying dependency graphs to tree structures in order to facilitate both the structure learning and optimization of the acquisition function. For the former, we propose a hybrid graph learning algorithm based on Gibbs sampling and mutation. In addition, we propose a novel zooming-based algorithm that permits generalized additive models to be employed more efficiently in the case of continuous domains. We demonstrate and discuss the efficacy of our approach via a range of experiments on synthetic functions and real-world datasets.

Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?

“…However, this is only possible if we know the decomposition of the black box. Existing methods attempt to learn it from data using maximum likelihood (Kandasamy et al, 2015;Rolland et al, 2018;Han et al, 2021), but there are no guarantees regarding the correctness of such a learning procedure. We expand on the problems associated with this approach in the next section.…”

“…challenging. Among the many proposed approaches ranging from linear to non-linear projections (Wang et al, 2016;Rana et al, 2017;Li et al, 2018;Tripp et al, 2020;Moriconi et al, 2020;Eriksson & Jankowiak, 2021;Grosnit et al, 2021b;Wan et al, 2021), decomposition methods that assume additively structured black-boxes emerged as a promising direction for high-d BO (Kandasamy et al, 2015;Rolland et al, 2018;Han et al, 2021).…”

“…Prior art empirically demonstrated that tree-structured decompositions (i.e., cycle-free pair-wise dimensional interactions) could effectively represent many black-box functions. The Tree algorithm (Han et al, 2021) discovers the best tree decomposition for the black-box based on the data collected during BO. In this situation, typical methods maximise a new GP marginal that includes both decomposition parameters (e.g., sparsity patterns, number of edges, etc.)…”

Monte Carlo Tree Search based Variable Selection for High Dimensional Bayesian Optimization

“…Wang et al [41] proposed ensemble BO that uses an ensemble of additive GP models for scalability. Han et al [13] constrained the dependency graphs of decomposition to tree structures to facilitate the decomposition learning and optimization. For most problems, however, the decomposition is unknown, and also difficult to learn.…”

“…Recently, scaling BO to high-dimensional problems has received a lot of interest. Decompositionbased methods [13,15,17,26,31] assume that the high-dimensional function to be optimized has a certain structure, typically the additive structure. By decomposing the original high-dimensional function into the sum of several low-dimensional functions, they optimize each low-dimensional function to obtain the point in the high-dimensional space.…”