Domain adaptation and sample bias correction theory and algorithm for regression

Abstract

We present a series of new theoretical, algorithmic, and empirical results for domain adaptation and sample bias correction in regression. We prove that the discrepancy is a distance for the squared loss when the hypothesis set is the reproducing kernel Hilbert space induced by a universal kernel such as the Gaussian kernel. We give new pointwise loss guarantees based on the discrepancy of the empirical source and target distributions for the general class of kernel-based regularization algorithms. These bounds have a simpler form than previous results and hold for a broader class of convex loss functions not necessarily differentiable, including L q losses and the hinge loss. We also give finer bounds based on the discrepancy and a weighted feature discrepancy parameter. We extend the discrepancy minimization adaptation algorithm to the more significant case where kernels are used and show that the problem can be cast as an SDP similar to the one in the feature space. We also show that techniques from smooth optimization can be used to derive an efficient algorithm for solving such SDPs even for very high-dimensional feature spaces and large samples. We have implemented this algorithm and report the results of experiments both with artificial and real-world data sets demonstrating its benefits both for general scenario of adaptation and the more specific scenario of sample bias correction. Our results show that it can scale to large data sets of tens of thousands or more points and demonstrate its performance improvement benefits.

Adaptation Algorithm and Theory Based on Generalized Discrepancy

“…The discrepancy has several advantages over a measure such as the L 1 or total variation distance (Cortes and Mohri, 2013): it is a finer measure than the L 1 distance, it takes into account the loss function and the hypothesis set, it can be accurately estimated from finite samples for common hypothesis sets such as kernel-based ones, it is symmetric and verifies the triangle inequality. It further defines a distance in the case of an L p loss used with a universal kernel such as a Gaussian kernel.…”

“…Using q min instead of Q amounts to reweighting the loss on the training samples to minimize the discrepancy between the empirical distribution and P . Besides its theoretical motivation, this algorithm has been shown to outperform several other algorithms in a series of experiments carried out by (Cortes and Mohri, 2013). Observe that, by definition, the solution q min of discrepancy minimization is obtained by minimizing a maximum over all pairs of hypotheses, that is max h,h ∈H |L P (h, h ) − L q min (h, h )|.…”

“…Assume as in several previous studies (Mansour et al, 2009;Cortes and Mohri, 2013) that the standard algorithm selected by the learner is regularized risk minimization over the Hilbert space H induced by a PSD kernel K. This covers a broad family of algorithms frequently used in applications. Ideally, that is in the absence of a domain adaptation problem, the learner would have access to the labels of the points in T .…”

“…Theorem 2 ( (Cortes and Mohri, 2013)) Let q be an arbitrary distribution over S X and let h * and h q be the hypotheses minimizing λ h 2…”

Estimation of Air Surface Temperature From Remote Sensing Images and Pixelwise Modeling of the Estimation Uncertainty Through Support Vector Machines

“…This behavior is generally shared by all supervised approaches to bio/geophysical parameter retrieval and suggests that, before applying the trained estimator to geographical regions distinct from those where the training samples were located, additional testing with data collected from those regions is expected to be necessary. The combination with domain adaptation techniques (e.g., [67]) could also be an interesting extension aimed at favoring application to areas without training samples.…”