Hashing-Based-Estimators for Kernel Density in High Dimensions

Abstract

Given a set of points P ⊂ d and a kernel k, the Kernel Density Estimate at a point x ∈ d is defined as KDE P (x) 1 |P| y∈P k(x, y). We study the problem of designing a data structure that given a data set P and a kernel function, returns approximations to the kernel density of a query point in sublinear time. We introduce a class of unbiased estimators for kernel density implemented through locality-sensitive hashing, and give general theorems bounding the variance of such estimators. These estimators give rise to efficient data structures for estimating the kernel density in high dimensions for a variety of commonly used kernels. Our work is the first to provide data-structures with theoretical guarantees that improve upon simple random sampling in high dimensions.

Multi-resolution Hashing for Fast Pairwise Summations

“…This corollary highlights the main point of our paper: we provide a general technique that enables the design of data structures that solve a variety of pairwise integration problems. For the special case of the Gaussian kernel for points on a sphere, our data structure has the same dependence in ε, r (up to poly-logarithmic factors in n) as the currently best known algorithm [22]. Extensions.…”

“…At a high level, we significantly generalize the recent approach of Hashing-Based-Estimators [22] to handle more general functions. This is done by combining classical ideas from Harmonic analysis (partitions of unity and approximation theory) with recent results for similarity search.…”

“…The main conceptual contribution of this work is a new framework for approximating pairwise summations. Our framework is based on a class of estimators that we introduce, called Multiresolution Hashing-Based-Estimators, that significantly generalizes previous work [22]. The main idea is that, instead of a single hashing scheme, we have a collection of hash families H t for t ∈ [T], where each H t is responsible for a different portion of the angular range around the query; H t has relatively high collision probability within the range assigned to it and relatively low outside.…”

“…Hashing-Based-Estimators (HBE). In a previous work of the authors [22], the general approach of using hashing to create importance sampling schemes with provable low-variance was introduced under the name of Hashing-Based-Estimators. Given a single hashing scheme H with collision probability p(x, y) h∼H [h(x) h(y)] an unbiased estimator for Z w (y) is constructed through a two-step sampling process (corresponds to the T 1 case of Multi-Resolution HBE).…”

“…The simplest and extremely general approach to get such estimators is through uniform random sampling. Letting χ ∈ (0, 1) be an upper bound on the failure probability, a second moment argument shows that storing and querying a uniform random sample of size O 1 ε 2 1 τ log(1/χ) is sufficient and necessary in general [48,22], to approximate the sum µ Z w (y) for any µ τ. The dependence on ε, χ is standard and easily shown to be necessary, so the question is for which class of functions can we improve the dependence on τ?…”