Randomized sketches for kernels: Fast and optimal nonparametric regression

Abstract

Kernel ridge regression (KRR) is a standard method for performing non-parametric regression over reproducing kernel Hilbert spaces. Given n samples, the time and space complexity of computing the KRR estimate scale as O(n 3 ) and O(n 2 ) respectively, and so is prohibitive in many cases. We propose approximations of KRR based on m-dimensional randomized sketches of the kernel matrix, and study how small the projection dimension m can be chosen while still preserving minimax optimality of the approximate KRR estimate. For various classes of randomized sketches, including those based on Gaussian and randomized Hadamard matrices, we prove that it suffices to choose the sketch dimension m proportional to the statistical dimension (modulo logarithmic factors). Thus, we obtain fast and minimax optimal approximations to the KRR estimate for non-parametric regression.

Statistically and Computationally Efficient Variance Estimator for Kernel Ridge Regression

“…The definition of V 2 is also our contribution. The variance V 2 is different from the variance that could be derived from the results in [15], which is:…”

“…Our work differs from [15] in several fundamental ways. [15] focuses on the (mean) prediction error on a training set, and it is unclear how this relates to a prediction error on a testing set.…”

“…Our work differs from [15] in several fundamental ways. [15] focuses on the (mean) prediction error on a training set, and it is unclear how this relates to a prediction error on a testing set. In contrast to [15], we target the variance of the prediction error, and focus on prediction on a test set.…”

“…Random matrices satisfying Assumption A1 include sub-Gaussian random matrices (see detailed proof in [8]), as well as matrices constructed by randomly sub-sampling and rescaling the rows of a fixed orthonormal matrix. We refer the interested reader to [15], [14] for more details.…”

“…An efficient way to break the computational bottleneck is low-rank approximation of kernel matrices. Existing methods include dimension reduction ( [16], [3], [2]), Nyström approximation ( [12], [1], [10]), and random projections of large kernel matrices ( [15]). Indeed, low-rank approximation strategies effectively reduce the size of large matrices such that the reduced matrices can be conveniently stored and processed.…”