Improved Coresets for Kernel Density Estimates

Abstract

We study the construction of coresets for kernel density estimates. That is we show how to approximate the kernel density estimate described by a large point set with another kernel density estimate with a much smaller point set. For characteristic kernels (including Gaussian and Laplace kernels), our approximation preserves the L ∞ error between kernel density estimates within error ε, with coreset size 4/ε 2 , but no other aspects of the data, including the dimension, the diameter of the point set, or the bandwidth of the kernel common to other approximations. When the dimension is unrestricted, we show this bound is tight for these kernels as well as a much broader set.This work provides a careful analysis of the iterative Frank-Wolfe algorithm adapted to this context, an algorithm called kernel herding. This analysis unites a broad line of work that spans statistics, machine learning, and geometry.When the dimension d is constant, we demonstrate much tighter bounds on the size of the coreset specifically for Gaussian kernels, showing that it is bounded by the size of the coreset for axis-aligned rectangles. Currently the best known constructive bound is O( 1 ε log d 1 ε ), and non-constructively, this can be improved by log 1 ε . This improves the best constant dimension bounds polynomially for d ≥ 3.

Multi-resolution Hashing for Fast Pairwise Summations

“…Gaussian, Laplacian). Furthermore, corresponding lower bounds Ω( √ d/ε) and Ω(1/ε 2 ) were proved for d 1/ε 2 [57] and d 1/ε 2 [56] respectively.…”

“…The literature has mostly been focused on obtaining additive error ε > 0. A general upper bound of O( 1 ε 2 ) was shown [16,56] on coresets for characteristic kernels (e.g. Gaussian, Laplacian) using a greedy construction (kernel herding [25]).…”

“…Gaussian, Laplacian) using a greedy construction (kernel herding [25]). The other approach [55,56,57] applies to Lipschitz kernels of bounded influence (decay fast enough), and constructs the core-set by starting with the full set of points and reducing it by half each time. Using smoothness properties of the kernel one can then bound the error introduced by each such operation through the notion of discrepancy [24,23].…”

Efficient Interpolation of Density Estimators

“…For example, Backurs et al [2019] show that for the Laplace and Exponential kernels with bandwidth h = 1, e.g., the value f (y) can be computed with multiplicative 1 ± ε error in time O( d √ τ ε 2 ) even in worst case over the dataset, where τ is a uniform lower bound on the KDE. Another effective approach to this problem in high dimensions is through coresets [Agarwal et al, 2005, Clarkson, 2010, Phillips and Tai, 2018a…”

Efficient construction of approximate ad-hoc ML models through materialization and reuse

“…Coresets were originally proposed in computational geometry that provide strong approximation guarantees. There has been extensive work on coresets for various ML models such as K-Means [8,25,18,48], GMM [17], kernel density estimation [46], logistic regression [28,24], SVM [10,52,51], Bayesian networks [42] and so on. A recent work [56] also used the concept for coresets.…”