A theory of learning from different domains

Abstract

Discriminative learning methods for classification perform well when training and test data are drawn from the same distribution. Often, however, we have plentiful labeled training data from a source domain but wish to learn a classifier which performs well on a target domain with a different distribution and little or no labeled training data. In this work we investigate two questions. First, under what conditions can a classifier trained from source data be expected to perform well on target data? Second, given a small amount of labeled target data, how should we combine it during training with the large amount of labeled source data to achieve the lowest target error at test time?Editors: Nicolo Cesa-Bianchi, David R. Hardoon, and Gayle Leen. Preliminary versions of the work contained in this article appeared in Advances in Neural InformationProcessing Systems (Ben-David et al. 2006;Blitzer et al. 2007a Learn (2010) 79: 151-175 We address the first question by bounding a classifier's target error in terms of its source error and the divergence between the two domains. We give a classifier-induced divergence measure that can be estimated from finite, unlabeled samples from the domains. Under the assumption that there exists some hypothesis that performs well in both domains, we show that this quantity together with the empirical source error characterize the target error of a source-trained classifier.We answer the second question by bounding the target error of a model which minimizes a convex combination of the empirical source and target errors. Previous theoretical work has considered minimizing just the source error, just the target error, or weighting instances from the two domains equally. We show how to choose the optimal combination of source and target error as a function of the divergence, the sample sizes of both domains, and the complexity of the hypothesis class. The resulting bound generalizes the previously studied cases and is always at least as tight as a bound which considers minimizing only the target error or an equal weighting of source and target errors.

Transfer Learning Through Greedy Subset Selection

“…In the literature there are several transfer learning settings [21,2,22,11]. We focus on the Hypothesis Transfer Learning framework (HTL, [15,3]).…”

“…While featuring good practical performance [11], and well understood theoretical guarantees [2], they often demonstrate poor scalability w.r.t. number of sources.…”

Extracting Explicit and Implicit Causal Relations from Sparse, Domain-Specific Texts

“…This is obtained by querying the Yahoo! search engine 3 with q= "y p ref x" (e.g. "flooding caused by rain"), and determining the fraction of the top-50 search results that contains phrase q in their summaries.…”

“…We then use these patterns to extract causal relations from domainspecific documents. Our strategy of applying the knowledge acquired from Wikipedia to specialized documents is based on domain-adaptation [3]. It circumvents the data sparsity issues posed by the domain-specific, corporate documents.…”

Adapting Component Analysis

“…In last decades, attention has been focused on domain adaptation problem in machine 1 A typical example is that of credit scoring: the training data set consists of only customers who have requested for a loan and their request has been accepted (the customers whose requests have been rejected are not included in the training data set).…”