On the diffusion approximation of nonconvex stochastic gradient descent

Hu, Wenqing; Li, Chris Junchi; Li, Lei; Liu, Jianguo

doi:10.48550/arxiv.1705.07562

Cited by 27 publications

(38 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A popular approach for analyzing SGD is based on considering SGD as a discretization of a continuous-time process [9,22,25,33,36,61]. This approach mainly requires the following assumption 1 on the stochastic gradient noise U k (w) ∇ fk (w) − ∇f (w):…”

Section: Introductionmentioning

confidence: 99%

A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

Şimşekli¹,

Sagun²,

Gürbüzbalaban³

2019

Preprint

View full text Add to dashboard Cite

The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kicks in. This assumption is often made for mathematical convenience, since it enables SGD to be analyzed as a stochastic differential equation (SDE) driven by a Brownian motion. We argue that the Gaussianity assumption might fail to hold in deep learning settings and hence render the Brownian motion-based analyses inappropriate. Inspired by non-Gaussian natural phenomena, we consider the GN in a more general context and invoke the generalized CLT (GCLT), which suggests that the GN converges to a heavy-tailed α-stable random variable. Accordingly, we propose to analyze SGD as an SDE driven by a Lévy motion. Such SDEs can incur 'jumps', which force the SDE transition from narrow minima to wider minima, as proven by existing metastability theory. To validate the α-stable assumption, we conduct extensive experiments on common deep learning architectures and show that in all settings, the GN is highly non-Gaussian and admits heavy-tails. We further investigate the tail behavior in varying network architectures and sizes, loss functions, and datasets. Our results open up a different perspective and shed more light on the belief that SGD prefers wide minima.

show abstract

Section: Introductionmentioning

confidence: 99%

A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

Şimşekli¹,

Sagun²,

Gürbüzbalaban³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…5 Considering (9), if these distributions are different, then the change in generalization error would become η tr Σ train /N + η G train • (G test − G train ), and thus has an additional piece that depends on the difference between test and train mean gradients. Since the second term is still present, this doesn't falsify our result (13), but shows that this divergence of gradients is also an important factor in overfitting.…”

Section: Various Commentsmentioning

confidence: 45%

“…The prevailing explanation given for this empirical observation has to do with descriptions of the loss landscape [8] (also see e.g. [9,[11][12][13][14][15]), in particular that SGD prefers "flat" to "sharp" minima in which the loss doesn't change too much in the neighborhood of the minima. An argument originating from [16] ascribes good generalization properties to such flat minima, reasoning that the effect of changing inputs could be reinterpreted as shifting the location of the minima.…”

Section: Introductionmentioning

confidence: 99%

“…suggesting that it's anisotropy that is important [20], and [15] argues that anisotropic noise is necessary for escaping from sharp minima. Either way, most of these analyses rely on taking the continuum limit [12,13,[20][21][22][23] in a way that is not necessarily justified, 2 and moreover assume that the covariance of the gradient does not depend strongly on the value of the model parameters. As a result, this leads many authors to conclude that some kind diffusion process in parameter space must play a fundamental role in finding minima that generalize well [12,22].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SGD Implicitly Regularizes Generalization Error

Roberts¹

2021

Preprint

View full text Add to dashboard Cite

We derive a simple and model-independent formula for the change in the generalization gap due to a gradient descent update. We then compare the change in the test error for stochastic gradient descent to the change in test error from an equivalent number of gradient descent updates and show explicitly that stochastic gradient descent acts to regularize generalization error by decorrelating nearby updates. These calculations depends on the details of the model only through the mean and covariance of the gradient distribution, which may be readily measured for particular models of interest. We discuss further improvements to these calculations and comment on possible implications for stochastic optimization.Note added: this paper appeared in the "Workshop on Integration of Deep Learning Theories" at NeurIPS in 2018 [1]. Given the current interest in this topic (see e.g. [2]), we would like to make the workshop paper more widely available. Other than this note and an updated affiliation, the paper is otherwise unchanged.

show abstract

“…A prolific technique for analyzing optimizations methods, both stochastic and deterministic, is the stochastic differential equations (SDE) paradigm [Chaudhari and Soatto, 2018, Hu et al, 2017, Jastrzebski et al, 2017, Kushner and Yin, 2003, Ljung, 1977, Mandt et al, 2016, Su et al, 2016. These SDEs relate to the dynamics of the optimization method by taking the limit when the stepsize goes to zero, so that the trajectory of the objective function over the lifetime of the algorithm converges to the solution of an SDE.…”

Section: Introductionmentioning

confidence: 99%

SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality

Paquette,

Lee,

Pedregosa

et al. 2021

Preprint

View full text Add to dashboard Cite

We propose a new framework, inspired by random matrix theory, for analyzing the dynamics of stochastic gradient descent (SGD) when both number of samples and dimensions are large. This framework applies to any fixed stepsize and the finite sum setting. Using this new framework, we show that the dynamics of SGD on a least squares problem with random data become deterministic in the large sample and dimensional limit. Furthermore, the limiting dynamics are governed by a Volterra integral equation. This model predicts that SGD undergoes a phase transition at an explicitly given critical stepsize that ultimately affects its convergence rate, which we also verify experimentally. Finally, when input data is isotropic, we provide explicit expressions for the dynamics and average-case convergence rates (i.e., the complexity of an algorithm averaged over all possible inputs). These rates show significant improvement over the worst-case complexities.

show abstract

On the diffusion approximation of nonconvex stochastic gradient descent

Cited by 27 publications

References 14 publications

A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

SGD Implicitly Regularizes Generalization Error

SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality

Contact Info

Product

Resources

About