Kernel selection with spectral perturbation stability of kernel matrix

Abstract

Kernel selection is one of the key issues both in recent research and application of kernel methods. This is usually done by minimizing either an estimate of generalization error or some other related performance measure. Use of notions of stability to estimate the generalization error has attracted much attention in recent years. Unfortunately, the existing notions of stability, proposed to derive the theoretical generalization error bounds, are difficult to be used for kernel selection in practice. It is well known that the kernel matrix contains most of the information needed by kernel methods, and the eigenvalues play an important role in the kernel matrix. Therefore, we aim at introducing a new notion of stability, called the spectral perturbation stability, to study the kernel selection problem. This proposed stability quantifies the spectral perturbation of the kernel matrix with respect to the changes in the training set. We establish the connection between the spectral perturbation stability and the generalization error. By minimizing the derived generalization error bound, we propose a new kernel selection criterion that can guarantee good generalization properties. In our criterion, the perturbation of the eigenvalues of the kernel matrix is efficiently computed by solving the derivative of a newly defined generalized kernel matrix. Both theoretical analysis and experimental results demonstrate that our criterion is sound and effective.

Automated Spectral Kernel Learning

“…Hyper parameters for kernels determine the performance of kernel methods. Meanwhile, kernel selection approaches have been well-studied (Liu and Liao 2014;Li et al 2017;Ding et al 2018;Liu et al 2018;. However, due to the separation of kernel selection and model training, those techniques are inefficient and lead the undesirable performance.…”

Eigenvalues Ratio for Kernel Selection of Kernel Methods

“…Minimizing theoretical estimate bounds of generalization error is an alternative to kernel selection. The widely used theoretical estimates usually introduce some measures of the complexity of the hypothesis space, such as VC dimension (Vapnik 2000), radius-margin bound (Vapnik 2000), maximal discrepancy (Bartlett, Boucheron, and Lugosi 2002), Rademacher complexity (Bartlett and Mendelson 2002), compression coefficient (Luxburg, Bousquet, and Schölkopf 2004), eigenvalues perturbation (Liu, Jiang, and Liao 2013), spectral perturbation stability (Liu and Liao 2014a), kernel stability (Liu and Liao 2014b) and covering number (Ding and Liao 2014). Unfortunately, for most of these measures, it is difficult to estimate their specific values (Nguyen and Ho 2007), hence hard to use them for kernel selection in practice.…”

Kernel semi-supervised graph embedding model for multimodal and mixmodal data

“…However, as the SGE model is a linear technique, it might not always give satisfying results in capturing the nonlinear structural characteristics of multimodal and mixmodal data. According to the kernel theory, when data is mapped nonlinearly with a kernel operator into a highdimensional dot product space, the nonlinear dimensionality reduction problems can be efficiently solved linearly [3,4]. This motivates us to ex-tend the SGE model to nonlinear case with the aid of kernel technique.…”