Covariate Shift by Kernel Mean Matching

Abstract

Given sets of observations of training and test data, we consider the problem of re-weighting the training data such that its distribution more closely matches that of the test data. We achieve this goal by matching covariate distributions between training and test sets in a high dimensional feature space (specifically, a reproducing kernel Hilbert space). This approach does not require distribution estimation. Instead, the sample weights are obtained by a simple quadratic programming procedure. We provide a uniform convergence bound on the distance between the reweighted training feature mean and the test feature mean, a transductive bound on the expected loss of an algorithm trained on the reweighted data, and a connection to single class SVMs. While our method is designed to deal with the case of simple covariate shift (in the sense of Chapter ??), we have also found benefits for sample selection bias on the labels. Our correction procedure yields its greatest and most consistent advantages when the learning algorithm returns a classifier/regressor that is "simpler" than the data might suggest.

Statistical analysis of kernel-based least-squares density-ratio estimation

“…. , X n without going through density estimation (Gretton et al 2009). The basic idea of KMM is to find w 0 (x) such that the mean discrepancy between nonlinearly transformed samples drawn from P and Q is minimized in a universal reproducing kernel Hilbert space (Steinwart 2001).…”

“…The kernel mean matching (KMM) method (Gretton et al 2009) directly gives estimates of the density ratio by matching the two distributions using universal reproducing kernel Hilbert spaces (Steinwart 2001). KMM can be regarded as a kernelized variant of Qin's moment matching estimator (Qin 1998).…”

Change-point detection in time-series data by relative density-ratio estimation

“…Thus, direct density-ratio estimation is substantially easier than density estimation . Following this idea, methods of direct density-ratio estimation have been developed , e.g., kernel mean matching (Gretton et al, 2009), the logistic-regression method (Bickel et al, 2007), and the Kullback-Leibler importance estimation procedure (KLIEP) (Sugiyama et al, 2008). In the context of change-point detection, KLIEP was reported to outperform other approaches (Kawahara and Sugiyama, 2012) such as the one-class support vector machine (Schölkopf et al, 2001;Desobry et al, 2005) and singular-spectrum analysis (Moskvina and Zhigljavsky, 2003b).…”

Adapting Component Analysis

“…Then an additional variable, s i , for each sample of training data set is defined [21,14]. s i is set to depend only on one of the sample features, therefore, the biasing procedure is called, simple bias [6]. This additional variable determines whether the corresponding sample is contributing in the biased training data set or not.…”

“…"Covariate shift" [5,6] and "class imbalance" [7] are two examples with different initial assumptions:…”