Mixed-Variable Bayesian Optimization

Abstract

<jats:p>The optimization of expensive to evaluate, black-box, mixed-variable functions, i.e. functions that have continuous and discrete inputs, is a difficult and yet pervasive problem in science and engineering. In Bayesian optimization (BO), special cases of this problem that consider fully continuous or fully discrete domains have been widely studied. However, few methods exist for mixed-variable domains and none of them can handle discrete constraints that arise in many real-world applications. In this paper, we introduce MiVaBo, a novel BO algorithm for the efficient optimization of mixed-variable functions combining a linear surrogate model based on expressive feature representations with Thompson sampling. We propose an effective method to optimize its acquisition function, a challenging problem for mixed-variable domains, making MiVaBo the first BO method that can handle complex constraints over the discrete variables. Moreover, we provide the first convergence analysis of a mixed-variable BO algorithm. Finally, we show that MiVaBo is significantly more sample efficient than state-of-the-art mixed-variable BO algorithms on several hyperparameter tuning tasks, including the tuning of deep generative models.</jats:p>

Black-box mixed-variable optimisation using a surrogate model that satisfies integer constraints

“…This problem has been addressed and solved for the continuous setting in the DONE algorithm [5] and for the discrete setting in the COMBO [27] and IDONE algorithms [6] by making use of parametric surrogate models that are linear in the parameters. The MiVaBO algorithm [9] is, to the best of our knowledge, the first algorithm that applies this solution to the mixed variable setting. It relies on an alternation between continuous and discrete optimisation to find the optimum of the surrogate model.…”

“…• choose them directly according to the data samples in a non-parametric way using kernel basis functions [21,25], • choose them randomly once and then fix them [4,5,9,27], or • choose them according to the variable domains , and then fix them [6].…”

“…If we would add basis functions that depend only on the continuous variables, the possible interaction between continuous and integer variables would not be modelled. But if we add randomly chosen mixed basis functions as in [9], then we might lose the guarantee that integer constraints are satisfied in local minima. See Figure 1 (left).…”

“…To avoid both problems, we propose to add mixed basis functions as in [9], but we choose them pseudo-randomly rather than randomly. This benefits from the success that randomly chosen weights have had in the past [4,5,9,27], while avoiding the problem from Figure 1 (left).…”

“…The second point ensures that the algorithm does not slow down over time, as the number of basis functions and free parameters of the surrogate model remains fixed. Finally, the last point eliminates the need for rounding techniques, and also eliminates the need for repeatedly using integer programming as is done in [9]. Though the model used by MVRSM does not guarantee that integer constraints are satisfied everywhere, it is constructed in such a way that these constraints are satisfied in the local optima of the model.…”