A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning

Abstract

The rapid recent progress in machine learning (ML) has raised a number of scientific questions that challenge the longstanding dogma of the field. One of the most important riddles is the good empirical generalization of overparameterized models. Overparameterized models are excessively complex with respect to the size of the training dataset, which results in them perfectly fitting (i.e., interpolating) the training data, which is usually noisy. Such interpolation of noisy data is traditionally associated with detrimental overfitting, and yet a wide range of interpolating models -from simple linear models to deep neural networks -have recently been observed to generalize extremely well on fresh test data. Indeed, the recently discovered double descent phenomenon has revealed that highly overparameterized models often improve over the best underparameterized model in test performance.Understanding learning in this overparameterized regime requires new theory and foundational empirical studies, even for the simplest case of the linear model. The underpinnings of this understanding have been laid in very recent analyses of overparameterized linear regression and related statistical learning tasks, which resulted in precise analytic characterizations of double descent. This paper provides a succinct overview of this emerging theory of overparameterized ML (henceforth abbreviated as TOPML) that explains these recent findings through a statistical signal processing perspective. We emphasize the unique aspects that define the TOPML research area as a subfield of modern ML theory and outline interesting open questions that remain.

Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inference

“…Section 3 showed how reducing the number of parameters (to a point) lowers the achievable membership advantage. However, as has been shown in a variety of works (e.g., [9,23,10]) increasing the number of parameters leads to a "double descent" behavior where generalization performance improves, as captured in the following proposition similar to Thm. 1 of [23] but specialized to the asymptotic case.…”

“…The general motivation for this paper is the recent realization that privacy issues around machine learning might be exacerbated by today's trend towards increasingly overparameterized models that have more parameters than training data points and so can be trained to memorize (attain zero error on) the training data. Surprisingly, some overparameterized models (e.g., massive deep networks [7,8]) generalize extremely well [9,10]. Limited empirical evidence suggests that overparameterization and memorization may lead to greater privacy vulnerabilities [11,12,13].…”

“…A related study by [12] analyzes the relationship between overfitting (as measured by generalization error) and membership inference. However, much recent work has shown that overparameterization does not always lead to increased generalization error and indeed sometimes can even decrease it [23,9,24,10], suggesting that more work is needed linking overparameterization and membership inference beyond mere generalization error. Our analysis is based on optimal membership inference attacks, a framework also employed by other studies of membership inference [12,25,13].…”

Deep Sequence Modeling for Pressure Controlled Mechanical Ventilation

“…Therefore, various methods were implemented, such as dropout and label smoothing, but conduct no obvious or even worsens the results. -Overparameterization of neural networks (Dar et al, 2021;Power et al, 2021), especially with a large number of epochs and heavy dropout rate, might be beneficial for long term fitting; -Multitask LSTM: adjust the model to predict secondary targets, thus adding additional targets to the original one to reduce overfitting and improve generalization. New targets such as pressure lags, cumulative pressure, and pressure variance at each timestep.…”

“…The method is feasible since there are only 950 possible output values in the training set; -Perform data augmentation, e.g., masking augmentation randomly performed on the R and C only, random shuffling of units within a specific short window in a sequence, and mixup augmentation in which two sequences are randomly selected and mixed. Other augmentations such as Cutout and CutMix can also be implemented; -Perform Fourier or Wavelets transforms to eliminate noise, create approximations of different values movements and generalize short and long-term trends; -Overparameterization of neural networks(Dar et al, 2021;Power et al, 2021), especially with a large number of epochs and heavy dropout rate, might be beneficial for long term fitting; -Multitask LSTM: adjust the model to predict secondary targets, thus adding additional targets to the original one to reduce overfitting and improve generalization. New targets such as pressure lags, cumulative pressure, and pressure variance at each timestep.…”