Balanced Distribution Adaptation for Transfer Learning

Abstract

Transfer learning has achieved promising results by leveraging knowledge from the source domain to annotate the target domain which has few or none labels. Existing methods often seek to minimize the distribution divergence between domains, such as the marginal distribution, the conditional distribution or both. However, these two distances are often treated equally in existing algorithms, which will result in poor performance in real applications. Moreover, existing methods usually assume that the dataset is balanced, which also limits their performances on imbalanced tasks that are quite common in real problems. To tackle the distribution adaptation problem, in this paper, we propose a novel transfer learning approach, named as Balanced Distribution Adaptation (BDA), which can adaptively leverage the importance of the marginal and conditional distribution discrepancies, and several existing methods can be treated as special cases of BDA. Based on BDA, we also propose a novel Weighted Balanced Distribution Adaptation (W-BDA) algorithm to tackle the class imbalance issue in transfer learning. W-BDA not only considers the distribution adaptation between domains but also adaptively changes the weight of each class. To evaluate the proposed methods, we conduct extensive experiments on several transfer learning tasks, which demonstrate the effectiveness of our proposed algorithms over several state-of-the-art methods.

Transfer Learning with Dynamic Adversarial Adaptation Network

“…4. It can be seen that the classification accuracy varies with different values of ω, which indicates the necessity to consider the different effects can be verified in BDA [8] and MEDA [7]), but also in adversarial transfer learning. Moreover, We find that the value of optimal ω varies on different tasks and even for the same task, ω may have several optimal values.…”

“…1), the local distribution should be given more attention. Two more recent work called Balanced Distribution Adaptation (BDA) [8] and Manifold Embedded Distribution Alignment (MEDA) [7] proposed to adaptively align these two distributions, while it is based on kernel method with high computational cost. In addition, MEDA is incapable of handling large-scale data.…”

Transfer Learning with Dynamic Distribution Adaptation

“…Others extended JDA by adding regularization [39], sparse representation [67], structural consistency [32], domain invariant clustering [55], and label propagation [74]. The work of Balanced Distribution Adaptation (BDA) [62] firstly proposed to manually weight the two distributions. The main differences between DDA (MDDA) and these methods are: 1) These work treats the two distributions equally.…”

“…To cope with the difference in distributions between domains, existing works can be summarized into two main categories: (a) instance reweighting [16,68], which reuses samples from the source domain according to some weighting technique; and (b) feature matching, which either performs subspace learning by exploiting the subspace geometrical structure [20,25,52,64], or distribution alignment to reduce the marginal or conditional distribution divergence between domains [40,62,74]. Recently, the success of deep learning has dramatically increased the performance of transfer learning either via deep representation learning [8,31,41,66,70,78] or adversarial learning [22,23,38,51,75].…”

“…Existing works only attempt to align the marginal [37,45,58] or the conditional distributions [38,48]. Although recent advance has suggested that aligning both distributions will lead to better performance [40,41,62], they only give the two distributions equal weights, which fails to evaluate the relative importance of these two distributions. In their assumptions, both the marginal and conditional distributions are contributing equally to the domain divergence.…”