Group kernels for Gaussian process metamodels with categorical inputs

Abstract

Gaussian processes (GP) are widely used as a metamodel for emulating time-consuming computer codes. We focus on problems involving categorical inputs, with a potentially large number L of levels (typically several tens), partitioned in G L groups of various sizes. Parsimonious covariance functions, or kernels, can then be defined by block covariance matrices T with constant covariances between pairs of blocks and within blocks. We study the positive definiteness of such matrices to encourage their practical use. The hierarchical group/level structure, equivalent to a nested Bayesian linear model, provides a parameterization of valid block matrices T. The same model can then be used when the assumption within blocks is relaxed, giving a flexible parametric family of valid covariance matrices with constant covariances between pairs of blocks. The positive definiteness of T is equivalent to the positive definiteness of a smaller matrix of size G, obtained by averaging each block. The model is applied to a problem in nuclear waste analysis, where one of the categorical inputs is atomic number, which has more than 90 levels.

Bayesian optimization of variable-size design space problems

“…The first and most simple discrete kernel to be considered in this paper is the Compound Symmetry (CS), characterized by a single covariance value for any non-identical pair of inputs [29]:…”

“…This characteristic further limits the number of suitable applications for this particular covariance function. Finally, it is worth mentioning that Roustant et al have extended the CS kernel in order to model mixed-variable functions characterized by discrete variables with a large number of levels [29]. The underlying idea is to group levels with similar characteristics, thus allowing to compute the covariance between said groups rather than between the levels.…”

“…Similarly to what is proposed by Roustant et al [29] for the modeling of discrete variables with large number of levels, the global variable-size design space kernel can also be represented under the form of a symmetric block matrix,…”

The Kalai-Smorodinski solution for many-objective Bayesian optimization

“…For now, the six variables are taken as continuous. But these results could be extended by increasing the number of variables, including categorical variables affecting more profoundly the structure of the network, for instance relying on the work of Roustant et al (2018).…”

Kernels over Sets of Finite Sets using RKHS Embeddings, with Application to Bayesian (Combinatorial) Optimization

“…One of the outstanding features of Gaussian Process (GP) prediction, in particular, is its usability to design Bayesian Optimization (BO) algorithms (Moćkus et al, 1978;Jones et al, 1998;Frazier, 2018) and further sequential design strategies (Risk and Ludkovski, 2018;Binois et al, 2019;Bect et al, 2019). While in most usual BO and related contributions the focus is on continuous problems with vector-valued inputs, there has been a growing interest recently for GP-related modelling and BO in the presence of discrete and mixed discrete-continuous inputs (Kondor and Lafferty, 2002;Gramacy and Taddy, 2010;Fortuin et al, 2018;Roustant et al, 2018;Garrido-Merchan and Hernández-Lobato, 2018;Ru et al, 2019;Griffiths and Hernández-Lobato, 2019). Here we focus specifically on kernels dedicated to finite set-valued inputs and their application to GP modelling and BO, notably (but not only) in combinatorial optimization.…”