Uniform-in-time weak error analysis for stochastic gradient descent algorithms via diffusion approximation

Abstract

Diffusion approximation provides weak approximation for stochastic gradient descent algorithms in a finite time horizon. In this paper, we introduce new tools motivated by the backward error analysis of numerical stochastic differential equations into the theoretical framework of diffusion approximation, extending the validity of the weak approximation from finite to infinite time horizon. The new techniques developed in this paper enable us to characterize the asymptotic behavior of constant-step-size SGD algorithms near a local minimum around which the objective functions are locally strongly convex, a goal previously unreachable within the diffusion approximation framework. Our analysis builds upon a truncated formal power expansion of the solution of a Kolmogorov equation arising from diffusion approximation, where the main technical ingredient is uniform-in-time bounds controlling the long-term behavior of the expansion coefficient functions near the local minimum. We expect these new techniques to bring new understanding of the behaviors of SGD near local minimum and greatly expand the range of applicability of diffusion approximation to cover wider and deeper aspects of stochastic optimization algorithms in data science.

Analysis of Kinetic Models for Label Switching and Stochastic Gradient Descent

“…If F = 0 we obtain an equation in the original time with a higher order term, which is reminiscent of the stochastic modified equations (cf. [4,8])…”

Implicit Gradient Regularization

“…However, implicit regularization and backward error analysis has not been explored. Backward error analysis was used [54,55] to study stochastic gradient descent in the context of stochastic differential equations and diffusion equations for the study of convergence and adaptive learning schemes, but, to the best of our knowledge, it has not been used to explore implicit regularization in gradient descent.…”

Approximation to Stochastic Variance Reduced Gradient Langevin Dynamics by Stochastic Delay Differential Equations

“…Our motivations, contributions and methods. It was recently discovered in [44,43,29,31,55,36,27,60,30,14] that SGD algorithms can be (weakly) approximated by continuous time SDEs. These SDEs often offer much needed insight to the algorithms under considerations, for instance, the continuous time treatment allows applications of stochastic control theory to develop novel adaptive algorithms [64,66].…”