Incorporating expert prior in Bayesian optimisation via space warping

Abstract

Bayesian optimisation is a well-known sample-efficient method for the optimisation of expensive black-box functions. However when dealing with big search spaces the algorithm goes through several low function value regions before reaching the optimum of the function. Since the function evaluations are expensive in terms of both money and time, it may be desirable to alleviate this problem. One approach to subside this cold start phase is to use prior knowledge that can accelerate the optimisation. In its standard form, Bayesian optimisation assumes the likelihood of any point in the search space being the optimum is equal. Therefore any prior knowledge that can provide information about the optimum of the function would elevate the optimisation performance. In this paper, we represent the prior knowledge about the function optimum through a prior distribution. The prior distribution is then used to warp the search space in such a way that space gets expanded around the high probability region of function optimum and shrinks around low probability region of optimum. We incorporate this prior directly in function model (Gaussian process), by redefining the kernel matrix, which allows this method to work with any acquisition function, i.e. acquisition agnostic approach. We show the superiority of our method over standard Bayesian optimisation method through optimisation of several benchmark functions and hyperparameter tuning of two algorithms: Support Vector Machine (SVM) and Random forest.

$π$BO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimization

“…Baselines We empirically evaluate πBO against the most competitive approaches for priors over the optimum described in Section 2.3: BOPrO (Souza et al, 2021) and BO in Warped Space (BOWS) (Ramachandran et al, 2020). To contextualize the performance of πBO, we provide additional, simpler baselines: random sampling, sampling from the prior and BO with prior-based initial design.…”

“…Additionally, it is restricted to only one specific acquisition function. Ramachandran et al (2020) use the probability integral transform to warp the search space, stretching high-probability regions and shrinking others. While the approach is model-and acquisition function agnostic, it requires invertible priors, and does not empirically display the ability to recover from misleading priors.…”

“…Next, we study the performance of πBO against other state-of-the-art prior-guided approaches. To this end, we consider optimizing 5 hyperparameters of an MLP for classification on 6 different OpenML datasets (Vanschoren et al, 2014) and compare against BOPrO (Souza et al, 2021) and BOWS (Ramachandran et al, 2020). For minimizing confounding factors, we implement πBO and BOWS in HyperMapper (Nardi et al, 2019), the same framework that BOPrO runs on.…”

“…While there is a demand to increase knowledge injection possibilities to further the adoption of BO, the concept of encoding prior beliefs over the location of an optimum is still rather novel: while there are some initial works (Ramachandran et al, 2020;Souza et al, 2021), no approach exists so far that allows the integration of arbitrary priors and offers flexibility in the choice of acquisition function; theory is also lacking. We close this gap by introducing a novel, remarkably simple, approach for injecting arbitrary prior beliefs into BO that is easy to implement, agnostic to the surrogate model used and converges at standard BO rates for any choice of prior.…”

Why Do Machine Learning Practitioners Still Use Manual Tuning? A Qualitative Study

“…Promising work in this direction includes various methods for integrating prior knowledge into Bayesian optimization. This can be achieved by directly specifying priors about the location of the optimum [40,41,42,43], or structural priors, e.g., in the form of log-transformations of hyperparameters [44], monotonicity constraints [45], or warping of hyperparameters [46].…”