Covariate Shift Adaptation for Structured Regression With Frank–Wolfe Algorithms

Abstract

This paper concerns structured regression problems wherein the issue of covariate shift is addressed, which aims at reducing the discrepancy in training and test data distributions, using computationally efficient and sparse optimization principles. In particular, the projection-free Frank-Wolfe optimization algorithms are used to learn the importance weights and re-weight the training data in the context of covariate shift. To determine the unbiased estimates of the weights, Kullback-Leibler importance estimation procedure is used but its computational cost can be high since it is based on projected gradient optimization. Instead of using the standard Frank-Wolfe algorithm, we adapt its variants and propose away-steps Frank-Wolfe and pairwise Frank-Wolfe covariate shift algorithms to correct the covariate shift. The results highlight the improved computational efficiency and sparsity achieved while learning the importance weights on synthetic as well as benchmark datasets. Furthermore, importance weighted Sharma-Mittal twin Gaussian process structured regression framework is proposed to incorporate the learned weights from covariate shift algorithms, and its equations are derived for importance weighted derivatives and uncertainties. The performance of proposed algorithms is evaluated on two applications of structured regression, namely, human pose estimation and music mood estimation, where the benefit of handling covariate shift is demonstrated with improved performance relative to the state-of-the-art techniques.

Impact of Regressand Stratification in Dataset Shift Caused by Cross-Validation

“…Target shift [14,15], which affects the distributions of the output variable P tra (y) = P tst (y), but it maintains the conditional distributions P tra (x|y) = P tst (x|y); 2. Covariate shift [35,36], which affects the distributions of the input attributes P tra (x) = P tst (x), but it maintains the conditional distributions P tra (y|x) = P tst (y|x).…”