Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks

Abstract

We study the transfer learning process between two linear regression problems. An important and timely special case is when the regressors are overparameterized and perfectly interpolate their training data. We examine a parameter transfer mechanism whereby a subset of the parameters of the target task solution are constrained to the values learned for a related source task. We analytically characterize the generalization error of the target task in terms of the salient factors in the transfer learning architecture, i.e., the number of examples available, the number of (free) parameters in each of the tasks, the number of parameters transferred from the source to target task, and the correlation between the two tasks. Our nonasymptotic analysis shows that the generalization error of the target task follows a two-dimensional double descent trend (with respect to the number of free parameters in each of the tasks) that is controlled by the transfer learning factors. Our analysis points to specific cases where the transfer of parameters is beneficial.

A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning

“…However, TOPML goes well beyond the linear regression task. Overparameterization naturally arises in diverse ML tasks, such as classification (e.g., Muthukumar et al, 2020a), subspace learning for dimensionality reduction (Dar et al, 2020), data generation (Luzi et al, 2021), and dictionary learning for sparse representations (Sulam et al, 2020). In addition, overparameterization arises in various learning settings that are more complex than elementary fully supervised learning: unsupervised and semi-supervised learning (Dar et al, 2020), transfer learning (Dar and Baraniuk, 2020), pruning of learned models (Chang et al, 2021), and others.…”

“…The study by Dar et al (2020) on overparameterized subspace learning was one of the first to explore interpolation phenomena beyond the realm of regression and classification. Dar et al (2020) start from considering an overparameterized version of a linear subspace fitting problem, which is commonly addressed via principal component analysis (PCA) of the training data. Recall that PCA is an unsupervised task, in contrast to the supervised nature of regression and classification.…”

“…However, the rank overparameterization of Zhuo et al (2021) is not related to interpolation nor the number of training examples, but only means that the solution matrix has a higher rank than the true matrix in the data model. This is in contrast to the rank overparameterization defined by Dar et al (2020) for studying overparameterization in the sense of interpolating solutions for subspace learning problems.…”

“…Motivated by the lack of double descent phenomena in PCA-based subspace learning, Dar et al (2020) define a new family of subspace learning problems in order to study their generalization behavior in overparameterized settings. This new family of learning problems connects the PCAbased subspace fitting approach (which is unsupervised and has strict orthonormality constraints on the learned matrix columns) and a regression-based approach (which is fully supervised and does not have orthonormality constraints on the learned matrix).…”

“…This new family of learning problems connects the PCAbased subspace fitting approach (which is unsupervised and has strict orthonormality constraints on the learned matrix columns) and a regression-based approach (which is fully supervised and does not have orthonormality constraints on the learned matrix). Specifically, Dar et al (2020)…”