On the diffusion approximation of nonconvex stochastic gradient descent

Hu, Wenqing; Li, Chris Junchi; Li, Lei; Liu, Jianguo

doi:10.4310/amsa.2019.v4.n1.a1

Cited by 55 publications

(53 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…RBM-1 is in spirit similar to the stochastic gradient descent in machine learning ( [21,22]). Recently, there are some analysis of SGD in the mathematical viewpoints and applications to physical problems [44,45,46,47]. RBM-1 can be used both for simulating the evolution of the measure (1.5) or (1.4) and for sampling from the equilibrium state π.…”

mentioning

confidence: 99%

Random Batch Methods (RBM) for interacting particle systems

Jin

Liu

2020

Journal of Computational Physics

Self Cite

113

189

View full text Add to dashboard Cite

We develop Random Batch Methods for interacting particle systems with large number of particles. These methods use small but random batches for particle interactions, thus the computational cost is reduced from O(N 2 ) per time step to O(N ), for a system with N particles with binary interactions. On one hand, these methods are efficient Asymptotic-Preserving schemes for the underlying particle systems, allowing N -independent time steps and also capture, in the N → ∞ limit, the solution of the mean field limit which are nonlinear Fokker-Planck equations; on the other hand, the stochastic processes generated by the algorithms can also be regarded as new models for the underlying problems. For one of the methods, we give a particle number independent error estimate under some special interactions. Then, we apply these methods to some representative problems in mathematics, physics, social and data sciences, including the Dyson Brownian motion from random matrix theory, Thomson's problem, distribution of wealth, opinion dynamics and clustering. Numerical results show that the methods can capture both the transient solutions and the global equilibrium in these problems.

show abstract

mentioning

confidence: 99%

Random Batch Methods (RBM) for interacting particle systems

Jin

Liu

2020

Journal of Computational Physics

Self Cite

113

189

View full text Add to dashboard Cite

show abstract

“…approximates the SGD iteration with weak error O(η 2 ), where W is a Wiener process, · denotes the Euclidean 2-norm 1 , and where BB T = Σ. Dropping the small term proportional to η reduces the weak error to O(η) (Hu et al, 2017). This leads to the SDE…”

Section: Stochastic Gradient Descent In Continuous-timementioning

confidence: 99%

A Continuous-Time Analysis of Distributed Stochastic Gradient

Boffi

Slotine²

2020

Neural Computation

View full text Add to dashboard Cite

We analyze the effect of synchronization on distributed stochastic gradient algorithms. By exploiting an analogy with dynamical models of biological quorum sensing -where synchronization between agents is induced through communication with a common signal -we quantify how synchronization can significantly reduce the magnitude of the noise felt by the individual distributed agents and by their spatial mean. This noise reduction is in turn associated with a reduction in the smoothing of the loss function imposed by the stochastic gradient approximation. Through simulations on model non-convex objectives, we demonstrate that coupling can stabilize higher noise levels and improve convergence. We provide a convergence analysis for strongly convex functions by deriving a bound on the expected deviation of the spatial mean of the agents from the global minimizer for an algorithm based on quorum sensing, the same algorithm with momentum, and the Elastic Averaging SGD arXiv:1812.10995v3 [math.OC] 15 Sep 2019 (EASGD) algorithm. We discuss extensions to new algorithms which allow each agent to broadcast its current measure of success and shape the collective computation accordingly. We supplement our theoretical analysis with numerical experiments on convolutional neural networks trained on the CIFAR-10 dataset, where we note a surprising regularizing property of EASGD even when applied to the non-distributed case. This observation suggests alternative second-order in-time algorithms for non-distributed optimization that are competitive with momentum methods.

show abstract

“…In the context of data science, diffusion approximation has been used to gain insights into online PCA [20], entropy-SGD [36,37], and nonconvex optimization [21], to name just a few. Despite its effectiveness as a continuous analogy of stochastic numerical optimization algorithms, the range of applicability of diffusion approximation is significantly limited by its restricted validity in a finite time interval.…”

Section: Main Contribution: Long-time Weak Approximation For Sgd Via Sdementioning

confidence: 99%

“…Stochastic gradient descent (SGD) is a prototypical stochastic optimization algorithm widely used for solving large scale data science problems [1,2,3,4,5,6], not only for its scalability to large datasets, but also due to its surprising capability of identifying parameters of deep neural network models with better generalization behavior than adaptive gradient methods [7,8,9]. The past decade has witnessed growing interests in accelerating this simple yet powerful optimization scheme [10,11,12,13,14,15], as well as better understanding its dynamics, through the lens of either discrete Markov chains [16,17] or continuous stochastic differential equations [18,19,20,21]. This paper introduces new techniques into the theoretical framework of diffusion approximation, which provides weak approximation to SGD algorithms through the solution of a modified stochastic differential equation (SDE).…”

Section: Introductionmentioning

confidence: 99%

Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation

Feng

Gao

et al. 2020

Communications in Mathematical Sciences

Self Cite

View full text Add to dashboard Cite

Diffusion approximation provides weak approximation for stochastic gradient descent algorithms in a finite time horizon. In this paper, we introduce new tools motivated by the backward error analysis of numerical stochastic differential equations into the theoretical framework of diffusion approximation, extending the validity of the weak approximation from finite to infinite time horizon. The new techniques developed in this paper enable us to characterize the asymptotic behavior of constant-step-size SGD algorithms near a local minimum around which the objective functions are locally strongly convex, a goal previously unreachable within the diffusion approximation framework. Our analysis builds upon a truncated formal power expansion of the solution of a Kolmogorov equation arising from diffusion approximation, where the main technical ingredient is uniform-in-time bounds controlling the long-term behavior of the expansion coefficient functions near the local minimum. We expect these new techniques to bring new understanding of the behaviors of SGD near local minimum and greatly expand the range of applicability of diffusion approximation to cover wider and deeper aspects of stochastic optimization algorithms in data science.

show abstract

On the diffusion approximation of nonconvex stochastic gradient descent

Cited by 55 publications

References 12 publications

Random Batch Methods (RBM) for interacting particle systems

Random Batch Methods (RBM) for interacting particle systems

A Continuous-Time Analysis of Distributed Stochastic Gradient

Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation

Contact Info

Product

Resources

About