A mean field view of the landscape of two-layer neural networks

Song, Mei; Montanari, Andrea; Nguyen, Phan-Minh

doi:10.1073/pnas.1806579115

Cited by 476 publications

(699 citation statements)

References 18 publications

Supporting

Mentioning

663

Contrasting

Unclassified

Order By: Relevance

“…We take n = 10 noisy measurements of this signal. Example 5 is directly inspired by feature models 17 in several recent papers [7,8,52], as well as having philosophical connections to other recent papers [6,46]. Figure 3 shows the performance of the minimum-2 -norm interpolator on Example 5 as we increase the number of features.…”

Section: The Minimum-2 -Norm Interpolator Through the Fourier Lensmentioning

confidence: 98%

“…≥ 3-layer neural networks.Promising progress has been made in all of these areas, which we recap only briefly below. Regarding the first point, while the optimization landscape for deep neural networks is non-convex and complicated, several independent recent works (an incomplete list is [12][13][14][15][16][17][18]) have shown that overparameterization can make it more attractive, in the sense that optimization algorithms like stochastic gradient descent (SGD) are more likely to actually converge to a global minimum. These interesting insights are mostly unrelated to the question of generalization, and should be viewed as a coincidental benefit of overparameterization.Second, a line of recent work [19][20][21][22] characterizes the inductive biases of commonly used optimization algorithms, thus providing insight into the identity of the global minimum that is selected.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Harmless Interpolation of Noisy Data in Regression

Muthukumar¹,

Vodrahalli²,

Subramanian³

et al. 2020

IEEE J. Sel. Areas Inf. Theory

View full text Add to dashboard Cite

A continuing mystery in understanding the empirical success of deep neural networks is their ability to achieve zero training error and generalize well, even when the training data is noisy and there are more parameters than data points. We investigate this overparameterized regime in linear regression, where all solutions that minimize training error interpolate the data, including noise. We characterize the fundamental generalization (mean-squared) error of any interpolating solution in the presence of noise, and show that this error decays to zero with the number of features. Thus, overparameterization can be explicitly beneficial in ensuring harmless interpolation of noise. We discuss two root causes for poor generalization that are complementary in nature -signal "bleeding" into a large number of alias features, and overfitting of noise by parsimonious feature selectors. For the sparse linear model with noise, we provide a hybrid interpolating scheme that mitigates both these issues and achieves order-optimal MSE over all possible interpolating solutions. arXiv:1903.09139v2 [cs.LG] 9 Sep 2019 2. We provide a Fourier-theoretic interpretation of concurrent analyses [6-10] of the minimum 2 -norm interpolator.3. We show (Theorem 2) that parsimonious interpolators (like the 1 -minimizing interpolator and its relatives) suffer the complementary problem of overfitting pure noise.4. We construct two-step hybrid interpolators that successfully recover signal and harmlessly fit noise, achieving the order-optimal rate of test MSE among all interpolators (Proposition 1 and all its corollaries). Related workWe discuss prior work in three categories: a) overparameterization in deep neural networks, b) interpolation of high-dimensional data using kernels, and c) high-dimensional linear regression. We then recap work on overparameterized linear regression that is concurrent to ours. Recent interest in overparameterizationConventional statistical wisdom is that using more parameters in one's model than data points leads to poor generalization. This wisdom is corroborated in theory by worst-case generalization bounds on such overparameterized models following from VC-theory in classification [2] and ill-conditioning in least-squares regression [5]. It is, however, contradicted in practice by the notable recent trend of empirically successful overparameterized deep neural networks. For example, the commonly used CIFAR-10 dataset contains 60000 images, but the number of parameters in all the neural networks achieving state-of-the-art performance on CIFAR-10 is at least 1.5 million [4]. These neural networks have the ability to memorize pure noisesomehow, they are still able to generalize well when trained with meaningful data.Since the publication of this observation [4,11], the machine learning community has seen a flurry of activity to attempt to explain this phenomenon, both for classification and regression problems, in neural networks. The problem is challenging for three core reasons 2 :1. The optimization landscape for l...

show abstract

Section: The Minimum-2 -Norm Interpolator Through the Fourier Lensmentioning

confidence: 98%

mentioning

confidence: 99%

Harmless Interpolation of Noisy Data in Regression

Muthukumar¹,

Vodrahalli²,

Subramanian³

et al. 2020

IEEE J. Sel. Areas Inf. Theory

View full text Add to dashboard Cite

show abstract

“…We show that the neural network output in the large hidden-units and large SGD-iterates limit depends on paths of representative weights that go from input to output layer. This result is then used to show that, under suitable assumptions, the limit neural network seeks to minimize the limit objective function and achieve zero loss.Recently, law of large numbers and central limit theorems have been established for neural networks with a single hidden layer [10,30,43,48,49,50]. For a single hidden layer, one can directly study the weak convergence of the empirical measure of the parameters.…”

mentioning

confidence: 99%

Mean Field Analysis of Neural Networks: A Law of Large Numbers

Sirignano¹,

Spiliopoulos

2020

SIAM J. Appl. Math.

113

View full text Add to dashboard Cite

We analyze multi-layer neural networks in the asymptotic regime of simultaneously (A) large network sizes and (B) large numbers of stochastic gradient descent training iterations. We rigorously establish the limiting behavior of the multi-layer neural network output. The limit procedure is valid for any number of hidden layers and it naturally also describes the limiting behavior of the training loss. The ideas that we explore are to (a) take the limits of each hidden layer sequentially and (b) characterize the evolution of parameters in terms of their initialization. The limit satisfies a system of deterministic integro-differential equations. The proof uses methods from weak convergence and stochastic analysis. We show that, under suitable assumptions on the activation functions and the behavior for large times, the limit neural network recovers a global minimum (with zero loss for the objective function). IntroductionMachine learning, and in particular deep learning, has achieved immense practical success, revolutionizing fields such as image, text, and speech recognition. It is also increasingly being used in engineering, medicine, and finance. However, despite their success in practice, there is currently limited mathematical understanding of deep neural networks. This has motivated recent mathematical research on multi-layer learning models such as [39], [40], [41], [20], [21], [42], [49], [50], [43], and [48].Neural networks are nonlinear statistical models whose parameters are estimated from data using stochastic gradient descent (SGD) methods. Deep learning uses neural networks with many layers (i.e., "deep" neural networks), which produces a highly flexible, powerful and effective model in practice. Typically, a neural network with multiple layers between the input and the output layer is called a "deep" neural network, see for example [24]. We analyze multi-layer neural networks that have a fixed number of layers between the input and output layer, and where the number of hidden units in each layer becomes large.Applications of deep learning include image recognition (see [35] and [24]), facial recognition [59], driverless cars [6], speech recognition (see [35], [4], [36], and [60]), and text recognition (see [62] and [57]). Neural networks also find increasing more applications in engineering, robotics, medicine, and finance (see [37], [38], [58], [26], [47], [3], [51], [52], [53], and [54]).In this paper we characterize multi-layer neural networks in the asymptotic regime of large network sizes and large numbers of stochastic gradient descent iterations. We rigorously prove the limit of the neural network output as the number of hidden units increases to infinity. The proof relies upon weak convergence analysis for stochastic processes. The result can be considered a "law of large numbers" for the neural network's output when both the network size and the number of stochastic gradient descent steps grow to infinity. We show that the neural network output in the large hidden-units and large SGD-iterat...

show abstract

“…Hence one important question for approximation of a function is the stability of this process [HR17]. Another approach is to develop the meanfield equations to approximate the function space with a fewer equations that are easier to handle [MMN18]. Poggio et al have found a bound for the complexity of a neural networks with smooth (e.g.…”

Section: -15 Cone Bipolars 1 Beta Ganglion Cellmentioning

confidence: 99%

On Functions Computed on Trees

et al. 2019

View full text Add to dashboard Cite

Any function can be constructed using a hierarchy of simpler functions through compositions. Such a hierarchy can be characterized by a binary rooted tree. Each node of this tree is associated with a function which takes as inputs two numbers from its children and produces one output. Since thinking about functions in terms of computation graphs is getting popular we may want to know which functions can be implemented on a given tree. Here, we describe a set of necessary constraints in the form of a system of non-linear partial differential equations that must be satisfied. Moreover, we prove that these conditions are sufficient in both contexts of analytic and bit-valued functions. In the latter case, we explicitly enumerate discrete functions and observe that there are relatively few. Our point of view allows us to compare different neural network architectures in regard to their function spaces. Our work connects the structure of computation graphs with the functions they can implement and has potential applications to neuroscience and computer science. 2 R. FARHOODI, K. FILOM, I. JONES, K. KORDING 7. Application to neural networks 42 7.1. From neural networks to trees 42 7.2. Trees with repeated labels; a toy example 43 7.3. Trees with repeated labels; estimates 46 References 49

show abstract

A mean field view of the landscape of two-layer neural networks

Cited by 476 publications

References 18 publications

Harmless Interpolation of Noisy Data in Regression

Harmless Interpolation of Noisy Data in Regression

Mean Field Analysis of Neural Networks: A Law of Large Numbers

On Functions Computed on Trees

Contact Info

Product

Resources

About