Sparsifying to optimize over multiple information sources: an augmented Gaussian process based algorithm

Abstract

Optimizing a black-box, expensive, and multi-extremal function, given multiple approximations, is a challenging task known as multi-information source optimization (MISO), where each source has a different cost and the level of approximation (aka fidelity) of each source can change over the search space. While most of the current approaches fuse the Gaussian processes (GPs) modelling each source, we propose to use GP sparsification to select only “reliable” function evaluations performed over all the sources. These selected evaluations are used to create an augmented Gaussian process (AGP), whose name is implied by the fact that the evaluations on the most expensive source are augmented with the reliable evaluations over less expensive sources. A new acquisition function, based on confidence bound, is also proposed, including both cost of the next source to query and the location-dependent approximation of that source. This approximation is estimated through a model discrepancy measure and the prediction uncertainty of the GPs. MISO-AGP and the MISO-fused GP counterpart are compared on two test problems and hyperparameter optimization of a machine learning classifier on a large dataset.

Fair and Green Hyperparameter Optimization via Multi-objective and Multiple Information Source Bayesian Optimization

“…Finally, although BO has been extended to deal with multiobjectives (Svenson & Santner, 2016;Feliot et al, 2017;Yang et al, 2019;Iqbal et al, 2020;Daulton et al, 2020) as well as multiple fidelities and multiple information sources (Lam et al, 2015;Poloczek et al, 2017;Ghoreishi & Allaire, 2019;Candelieri & Archetti, 2021b;a;Ariafar et al, 2021), there is a significant lack of solutions jointly addressing the two tasks. On the other hand, the research interest on this specific challenge is quickly increasing, especially because its applicability to many other real-life problems than fair and green ML, as demonstrated by very recent works such as (Sun et al, 2022) and (Irshad et al, 2021).…”

“…BO has been also successfully extended to deal with MISO problems, where each information source, is individually modelled through a probabilistic surrogate model -usually a Gaussian Process (GP) -fitted on the queries performed on that source. Then, all the individual models are combined into a single one, which is used to drive the choice of the next promising source-location pair to query, such as in (Ghoreishi & Allaire, 2019;Candelieri & Archetti, 2021b).…”

“…Then, a single GP model is fitted, for each objective, by combining the GPs individually modelling that objective on every information source. In FanG-HPO this operation is performed by following the Augmented Gaussian Process (AGP) approach recently proposed in (Candelieri & Archetti, 2021b). More precisely, a set of indices identifying "reliable" observations from cheaper sources is computed for each source-objective pair, where "reliable" means they are not too discrepant with respect the ground-truth:…”

“…where α is a technical parameter to tune reliability of observations from cheap information sources (in (Candelieri & Archetti, 2021b) the suggested value is α = 1).…”

“…Thanks to its sample-efficiency, BO is the core component of most of the current Automated Machine Learning (AutoML) (Hutter et al, 2019;He et al, 2021) solutions, both open-source and commercial. BO has been recently extended to also deal with multiple objectives (Hernández-Lobato et al, 2016;Paria et al, 2020), as well as multiple information sources with different computational cost can be accessed (Ghoreishi & Allaire, 2019;Belakaria et al, 2020a;Candelieri et al, 2021;Candelieri & Archetti, 2021b;Khatamsaz et al, 2020). A special case is when sources can be organized hierarchically depending on their quality of approximation (aka fidelity), leading to the so-called multifidelity optimization, originally proposed in (Kennedy & O'Hagan, 2000).…”