Faster Kernel Ridge Regression Using Sketching and Preconditioning

Abstract

Kernel Ridge Regression is a simple yet powerful technique for non-parametric regression whose computation amounts to solving a linear system. This system is usually dense and highly ill-conditioned. In addition, the dimensions of the matrix are the same as the number of data points, so direct methods are unrealistic for large-scale datasets. In this paper, we propose a preconditioning technique for accelerating the solution of the aforementioned linear system. The preconditioner is based on random feature maps, such as random Fourier features, which have recently emerged as a powerful technique for speeding up and scaling the training of kernel-based methods, such as kernel ridge regression, by resorting to approximations. However, random feature maps only provide crude approximations to the kernel function, so delivering state-of-the-art results by directly solving the approximated system requires the number of random features to be very large. We show that random feature maps can be much more effective in forming preconditioners, since under certain conditions a not-too-large number of random features is sufficient to yield an effective preconditioner. We empirically evaluate our method and show it is highly effective for datasets of up to one million training examples.

Data-driven Random Fourier Features using Stein Effect

“…As long as M is sufficiently smaller than n, this leads to more scalable solutions, e.g., for regression we get back to O(nM 2 ) training and O(M d) prediction time, with O(nM ) memory requirements. We can also apply random features to unsupervised learning problems, such as kernel clustering [Chitta et al, 2012].…”

“…An important question is how to construct different weights on the integrand functions to have a better kernel approximation. Given a fixed QMC sequence, Avron et al [2016b] solve the weights by optimizing a derived error bound based on the observed data. There are two potential weakness of [Avron et al, 2016b].…”

“…Given a fixed QMC sequence, Avron et al [2016b] solve the weights by optimizing a derived error bound based on the observed data. There are two potential weakness of [Avron et al, 2016b]. First, Avron et al [2016b] does not fully utilize the data information because the optimization objective only depends on the upper bound information of |x i | instead of Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence the distribution of x i .…”

“…There are two potential weakness of [Avron et al, 2016b]. First, Avron et al [2016b] does not fully utilize the data information because the optimization objective only depends on the upper bound information of |x i | instead of Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence the distribution of x i . Second, the error bound used in Avron et al [2016b] is potentially loose.…”

“…Such a treatment, however, is arguably sub-optimal for minimizing the expected risk in kernel approximation. Avron et al [2016b] presented a weighting strategy for minimizing a loose error bound which depends on the maximum norm of data points. On the other hand, Bayesian Quadrature (BQ) is a powerful framework that solves the integral approximation by E[f (x)] ≈ m β m f (x m ) where f (x m ) is feature function and β m is with non-uniform weights.…”