Deep learning: a statistical viewpoint

Abstract

The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting, that is, accurate predictions despite overfitting training data. In this article, we survey recent progress in statistical learning theory that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behavior of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favorable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning

“…In this paper, we survey the emerging field of TOPML research with a principal focus on foundational principles developed in the past few years. Compared to other recent surveys (Bartlett et al, 2021;Belkin, 2021), we take a more elementary signal processing perspective to elucidate these principles. Formally, we define the TOPML research area as the sub-field of ML theory where 1. there is clear consideration of exact or near interpolation of training data 2. the learned model complexity is high with respect to the training dataset size.…”

“…≥ λ p . We refer the reader to Bartlett et al (2021) for a detailed exposition of these results and their consequences. Here, we present the essence of these results in a popular model used in high-dimensional statistics, the spiked covariance model.…”

Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent

“…These empirical mysteries inspire a recent flurry of activity towards understanding the generalization properties of various interpolators. A dominant fraction of recent efforts, however, concentrated on studying certain minimum 2 -norm interpolators, primarily in the context of linear and/or kernel regression (see, e.g., ; Mei and Montanari (2019); Hastie et al (2019); Belkin et al (2020); Bartlett et al (2020Bartlett et al ( , 2021 and the references therein). This was in part due to the existence of closed-form expressions for minimum 2 -norm interpolators, which are particularly handy when determining the statistical risk.…”

On generalization bounds for deep networks based on loss surface implicit regularization

“…Obstacles in the theoretical foundation include the higher-order nonlinear structures due to the stacking of multiple layers and the excessive number of network parameters in state of the art networks. For some recent surveys, see [5,6].…”

“…For the filtration {F 𝑡 } 𝑡≥0 introduced before, batch gradient noise is defined as the A-dependent F 𝑡+1measurable random vector 𝑊 𝑡+1 (A) = √ 𝑚(∇𝑅 𝑛 (A) − ∇ 𝑅 𝑡 𝑛 (A)). This random variable measures the effect of the subsampling on the gradient and allows to rewrite (5) as…”