A Hilbert Space Embedding for Distributions

Abstract

Abstract. We describe a technique for comparing distributions without the need for density estimation as an intermediate step. Our approach relies on mapping the distributions into a reproducing kernel Hilbert space. Applications of this technique can be found in two-sample tests, which are used for determining whether two sets of observations arise from the same distribution, covariate shift correction, local learning, measures of independence, and density estimation.Kernel methods are widely used in supervised learning [1,2,3,4], however they are much less established in the areas of testing, estimation, and analysis of probability distributions, where information theoretic approaches [5,6] have long been dominant. Recent examples include [7] in the context of construction of graphical models, [8] in the context of feature extraction, and [9] in the context of independent component analysis. These methods have by and large a common issue: to compute quantities such as the mutual information, entropy, or Kullback-Leibler divergence, we require sophisticated space partitioning and/or bias correction strategies [10,9].In this paper we give an overview of methods which are able to compute distances between distributions without the need for intermediate density estimation. Moreover, these techniques allow algorithm designers to specify which properties of a distribution are most relevant to their problems. We are optimistic that our embedding approach to distribution representation and analysis will lead to the development of algorithms which are simpler and more effective than entropy-based methods in a broad range of applications.We begin our presentation in Section 1 with an overview of reproducing kernel Hilbert spaces (RKHSs), and a description of how probability distributions can be represented as elements in an RKHS. In Section 2, we show how these representations may be used to address a variety of problems, including homogeneity testing (Section 2.1), covariate shift correction (Section 2.2), independence measurement (Section 2.3), feature extraction (Section 2.4), and density estimation (Section 2.5).

Adaptive Change Detection for Long-Term Machinery Monitoring Using Incremental Sliding-Window

“…Proposed scheme: Inspired by Ref. [28], probabilistic distributions can be embedded in RKHS. The center of the HED are the mean mapping functions:…”

“…As long as the rademacher average [29], which measures the ''size'' of a class of real-valued functions with respect to a probability distribution, is well behaved, finite sample yield error converges to zero, thus they empirically approximate lðP x Þ(see Ref. [28] for more details).…”

Robust Density Comparison Using Eigenvalue Decomposition

“…In terms of the Hilbert space embedding, the density function estimate results from the inner product of the mapped point φ(u) with the mean of the distribution μ[P u ]. The mean map μ : Smola et al (2007), and allows for the definition of a similarity measure between two sampled sets P u and P v , sampled from the same or two different distributions. The measure is defined to be D (P u , P v …”

“…This makes the similarity measures, such as Kullback-Leibler divergence and Bhattacharyya coefficient, computationally unstable, Yang & R. Duraiswami (2005). Additionally, these techniques require sophisticated space partitioning and/or bias correction strategies Smola et al (2007).…”

A test of independence based on a generalized correlation function

“…Next, consider the measure based on reproducing kernel Hilbert space (RKHS) [34]. It has been shown that if a kernel κ is characteristic then there exist a unique and one to one mapping between the space of probability density functions and the mean operator µ in the RKHS.…”