Good practices for Bayesian optimization of high dimensional structured spaces

Abstract

The increasing availability of structured but high dimensional data has opened new opportunities for optimization. One emerging and promising avenue is the exploration of unsupervised methods for projecting structured high dimensional data into low dimensional continuous representations, simplifying the optimization problem and enabling the application of traditional optimization methods. However, this line of research has been purely methodological with little connection to the needs of practitioners so far. In this article, we study the effect of different search space design choices for performing Bayesian optimization in high dimensional structured datasets. In particular, we analyses the influence of the dimensionality of the latent space, the role of the acquisition function and evaluate new methods to automatically define the optimization bounds in the latent space. Finally, based on experimental results using synthetic and real datasets, we provide recommendations for the practitioners.

Structure-based hyperparameter selection with Bayesian optimization in multidimensional scaling

“…This approach works well for STOPS due to three aspects: First, BO needs only function evaluations so the modularity of STOPS and the lack of exploitable structure is no hindrance. Second, BO is competitive in situations where the parameter vector is low-dimensional (e.g., Siivola et al 2021) as is the case for all the θ we outlined in Sect. 4 (with at most 3).…”

Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?

“…However, over the course of this pretraining, we need to ensure that the obtained latent space is well-suited for BO. There have been numerous VAE-based algorithms proposed (Eissman et al, 2018;Gómez-Bombarelli et al, 2018;Zhang et al, 2019;Tripp et al, 2020;Griffiths & Hernández-Lobato, 2020;Siivola et al, 2021;Grosnit et al, 2021b), employing various mechanisms to achieve this goal.…”

Bayesian optimization in continuous spaces via virtual process embeddings

“…The effects of the value of hyperparameter l can be observed in the accompanying Jupyter notebook that runs the BO algorithm with 4 different values of l viz., 1, 2, 5, 10. The applications of BO algorithm are ubiquitous and can be found in a plethora of elds including but not limited to materials science, [38][39][40] catalysis, 41,42 3D printing, 43,44 chemistry, 20,45 solar cells, 5,46 pharmaceuticals, 47,48 While the application of BO in the latent space of the VAE is a recent development, 44,45,[49][50][51][52] the optimization in the latent space of trajectories where the trajectories in the input dataset are constructed for a domain specic application is still not explored.…”