Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimality

Abstract

This paper considers the estimation and prediction of a high-dimensional linear regression in the setting of transfer learning, using samples from the target model as well as auxiliary samples from different but possibly related regression models. When the set of "informative" auxiliary samples is known, an estimator and a predictor are proposed and their optimality is established. The optimal rates of convergence for prediction and estimation are faster than the corresponding rates without using the auxiliary samples. This implies that knowledge from the informative auxiliary samples can be transferred to improve the learning performance of the target problem. In the case that the set of informative auxiliary samples is unknown, we propose a data-driven procedure for transfer learning, called Trans-Lasso, and reveal its robustness to non-informative auxiliary samples and its efficiency in knowledge transfer. The proposed procedures are demonstrated in numerical studies and are applied to a dataset concerning the associations among gene expressions. It is shown that Trans-Lasso leads to improved performance in gene expression prediction in a target tissue by incorporating the data from multiple different tissues as auxiliary samples.

Transfer Learning under High-dimensional Generalized Linear Models

“…The algorithms we propose follow from the idea in Bastani (2020) and Li et al (2020a), which we call two-step transfer learning algorithms. The main idea is to first transfer the information from transferable sources to obtain a rough estimator, then correct the bias in the second step using the target data.…”

“…In this work, we contribute to the high-dimensional transfer learning framework in the following perspectives. First, we extend the results of Bastani (2020) and Li et al (2020a), by proposing transfer learning algorithms on generalized linear models (GLMs).…”

“…Bastani (2020) studied the single-source case when the target data follows a high-dimensional linear model while the source sample size is much larger than the dimension p. A two-step transfer learning algorithm was developed and the estimation error bound was proved. Li et al (2020a) further weakened the assumptions and explored the multi-source high-dimensional linear regression problem. The 2 -estimation error bound under q -regularization (q ∈ [0, 1]) was derived and proved to be minimax optimal under some conditions.…”

Targeting Underrepresented Populations in Precision Medicine: A Federated Transfer Learning Approach

“…Bastani (2020) studied estimation and prediction in high-dimensional linear models and the sample size of the auxiliary study is larger than the number of covariates. Li et al (2020a) propose a minimax optimal transfer learning algorithm in high-dimensional linear models and study the adaptation to the unknown similarity level. Li et al (2020b) studies transfer learning in high-dimensional Gaussian graphical models with false discovery rate control.…”

“…Compared to a recent work (Tian and Feng, 2021), the above procedure has a faster convergence rate, which is in fact minimax optimal under mild conditions. Moreover, our method learns w (k) independently in Step 1 and Step 2, while in other related methods (Tian and Feng, 2021;Li et al, 2020a), a pooled analysis is conducted with data from multiple populations. In a federated setting, finding a proper initialization is challenging for such a pooled estimator due to various levels of heterogeneity.…”