Eigenvalues Ratio for Kernel Selection of Kernel Methods

Lin

et al. 2017

AAAI

Self Cite

Kernel learning is a fundamental problem both in recent research and application of kernel methods. Existing kernel learning methods commonly use some measures of generalization errors to learn the optimal kernel in a convex (or conic) combination of prescribed basic kernels. However, the generalization bounds derived by these measures usually have slow convergence rates, and the basic kernels are finite and should be specified in advance. In this paper, we propose a new kernel learning method based on a novel measure of generalization error, called principal eigenvalue proportion (PEP), which can learn the optimal kernel with sharp generalization bounds over the convex hull of a possibly infinite set of basic kernels. We first derive sharp generalization bounds based on the PEP measure. Then we design two kernel learning algorithms for finite kernels and infinite kernels respectively, in which the derived sharp generalization bounds are exploited to guarantee faster convergence rates, moreover, basic kernels can be learned automatically for infinite kernel learning instead of being prescribed in advance. Theoretical analysis and empirical results demonstrate that the proposed kernel learning method outperforms the state-of-the-art kernel learning methods.

Section: Measures Of Generalization Errormentioning

confidence: 99%

Infinite Kernel Learning: Generalization Bounds and Algorithms

Liu

Lin

et al. 2017

AAAI

Self Cite

“…The upper bounds are composed of the error on data and the complexity of the hypothesis space (Bartlett, Boucheron, and Lugosi 2002). Different measures of the complexity constitute different kernel selection criteria, such as Rademacher complexity (Bartlett and Mendelson 2002), local Rademacher complexity (Cortes, Kloft, and Mohri 2013), radius-margin bound (Chapelle et al 2002), maximum mean discrepancy (MMD) (Sriperumbudur et al 2009;Gretton et al 2012a;2012b;Song et al 2012), effective dimensionality (Zhang 2005;Bach 2013), eigenvalues ratio (Liu and Liao 2015), and the covering number (Ding and Liao 2014b). The second category is to maximize the similarity between the kernel matrix and the label matrix.…”

Section: Introductionmentioning

confidence: 99%

Randomized Kernel Selection With Spectra of Multilevel Circulant Matrices

Ding

Liu

et al. 2018

AAAI

Self Cite

Kernel selection aims at choosing an appropriate kernel function for kernel-based learning algorithms to avoid either underfitting or overfitting of the resulting hypothesis. One of the main problems faced by kernel selection is the evaluation of the goodness of a kernel, which is typically difficult and computationally expensive. In this paper, we propose a randomized kernel selection approach to evaluate and select the kernel with the spectra of the specifically designed multilevel circulant matrices (MCMs), which is statistically sound and computationally efficient. Instead of constructing the kernel matrix, we construct the randomized MCM to encode the kernel function and all data points together with labels. We build a one-to-one correspondence between all candidate kernel functions and the spectra of the randomized MCMs by Fourier transform. We prove the statistical properties of the randomized MCMs and the randomized kernel selection criteria, which theoretically qualify the utility of the randomized criteria in kernel selection. With the spectra of the randomized MCMs, we derive a series of randomized criteria to conduct kernel selection, which can be computed in log-linear time and linear space complexity by fast Fourier transform (FFT). Experimental results demonstrate that our randomized kernel selection criteria are significantly more efficient than the existing classic and widely-used criteria while preserving similar predictive performance.

A survey on online kernel selection for online kernel learning

Zhang

2018

WIREs Data Min & Knowl

Online kernel selection is fundamental to online kernel learning. In contrast to offline kernel selection, online kernel selection intermixes kernel selection and training at each round of online kernel learning, and requires a sublinear regret bound and low computational complexity. In this paper, we first compare the difference between offline kernel selection and online kernel selection, then survey existing online kernel selection approaches from the perspectives of formulation, algorithm, candidate kernels, computational complexities and regret guarantees, and finally point out some future research directions in online kernel selection.