New Analysis and Algorithm for Learning with Drifting Distributions

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Medina

2015

Self Cite

We present a new algorithm for domain adaptation improving upon a discrepancy minimization algorithm previously shown to outperform a number of algorithms for this task. Unlike many previous algorithms for domain adaptation, our algorithm does not consist of a fixed reweighting of the losses over the training sample. We show that our algorithm benefits from a solid theoretical foundation and more favorable learning bounds than discrepancy minimization. We present a detailed description of our algorithm and give several efficient solutions for solving its optimization problem. We also report the results of several experiments showing that it outperforms discrepancy minimization.

Section: Introductionmentioning

confidence: 99%

Adaptation Algorithm and Theory Based on Generalized Discrepancy

Cortes

Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Medina

2015

Self Cite

“…If Z T 1 is a sequence of independent but not identically distributed random variables, we recover the results of Mohri and Muñoz (2012). In the i.i.d.…”

Section: Theorem 1 With the Assumptions Of Lemma 3 For Any δ >mentioning

confidence: 57%

“…A key ingredient of the bounds we present is the notion of discrepancy between two probability distributions that was used by Mohri and Muñoz (2012) to give generalization bounds for sequences of independent (but not identically distributed) random variables. In our setting, discrepancy can be defined as …”

Section: Introductionmentioning

confidence: 99%

Generalization bounds for non-stationary mixing processes

Kuznetsov

2016

Mach Learn

Self Cite

This paper presents the first generalization bounds for time series prediction with a non-stationary mixing stochastic process. We prove Rademacher complexity learning bounds for both average-path generalization with non-stationary β-mixing processes and path-dependent generalization with non-stationary φ-mixing processes. Our guarantees are expressed in terms of β-or φ-mixing coefficients and a natural measure of discrepancy between training and target distributions. They admit as special cases previous Rademacher complexity bounds for non-i.i.d. stationary distributions, for independent but not identically distributed random variables, or for the i.i.d. case. We show that, using a new sub-sample selection technique we introduce, our bounds can be tightened under the natural assumption of asymptotically stationary stochastic processes. We also prove that fast learning rates can be achieved by extending existing local Rademacher complexity analyses to the non-i.i.d. setting. We conclude the paper by providing generalization bounds for learning with unbounded losses and non-i.i.d. data.

“…Finally, the notion of discrepancy and its extensions similarly play a critical role in the analysis of learning with drifting distributions [22], a scenario that is closely related to those of domain adaption and sample bias correction.…”

Section: Resultsmentioning

confidence: 99%

“…Theorem 6 Let z * be an optimal solution for problem (22) and let v k be defined as in Algorithm 1, then for any [26]. This can be done by finding a uniform -approximation of F by a smooth convex function G with Lipschitz-continuous gradient.…”

Section: Solution Based On Smooth Approximationmentioning

confidence: 99%

Domain adaptation and sample bias correction theory and algorithm for regression

Cortes

Theoretical Computer Science

2014

Self Cite

118

112

We present a series of new theoretical, algorithmic, and empirical results for domain adaptation and sample bias correction in regression. We prove that the discrepancy is a distance for the squared loss when the hypothesis set is the reproducing kernel Hilbert space induced by a universal kernel such as the Gaussian kernel. We give new pointwise loss guarantees based on the discrepancy of the empirical source and target distributions for the general class of kernel-based regularization algorithms. These bounds have a simpler form than previous results and hold for a broader class of convex loss functions not necessarily differentiable, including L q losses and the hinge loss. We also give finer bounds based on the discrepancy and a weighted feature discrepancy parameter. We extend the discrepancy minimization adaptation algorithm to the more significant case where kernels are used and show that the problem can be cast as an SDP similar to the one in the feature space. We also show that techniques from smooth optimization can be used to derive an efficient algorithm for solving such SDPs even for very high-dimensional feature spaces and large samples. We have implemented this algorithm and report the results of experiments both with artificial and real-world data sets demonstrating its benefits both for general scenario of adaptation and the more specific scenario of sample bias correction. Our results show that it can scale to large data sets of tens of thousands or more points and demonstrate its performance improvement benefits.