What Happens after SGD Reaches Zero Loss? --A Mathematical Framework

Abstract

Understanding the implicit bias of Stochastic Gradient Descent (SGD) is one of the key challenges in deep learning, especially for overparametrized models, where the local minimizers of the loss function L can form a manifold. Intuitively, with a sufficiently small learning rate η, SGD tracks Gradient Descent (GD) until it gets close to such manifold, where the gradient noise prevents further convergence. In such a regime, Blanc et al. ( 2020) proved that SGD with label noise locally decreases a regularizer-like term, the sharpness of loss, tr[∇ 2 L]. The current paper gives a general framework for such analysis by adapting ideas from Katzenberger (1991). It allows in principle a complete characterization for the regularization effect of SGD around such manifold-i.e., the "implicit bias"-using a stochastic differential equation (SDE) describing the limiting dynamics of the parameters, which is determined jointly by the loss function and the noise covariance. This yields some new results: (1) a global analysis of the implicit bias valid for η −2 steps, in contrast to the local analysis of Blanc et al. ( 2020) that is only valid for η −1.6 steps and (2) allowing arbitrary noise covariance. As an application, we show with arbitrary large initialization, label noise SGD can always escape the kernel regime and only requires O(κ ln d) samples for learning an κ-sparse overparametrized linear model in R d (Woodworth et al., 2020), while GD initialized in the kernel regime requires Ω(d) samples. This upper bound is minimax optimal and improves the previous O(κ 2 ) upper bound (HaoChen et al., 2020).

The Multiscale Structure of Neural Network Loss Functions: The Effect on Optimization and Origin

“…which recovers the results in [15]. This quasistatic approach can be easily adapted to other types of noise and other optimizers such as SGD momentum.…”

“…Remark 1. The idea of flatness driven motion along the manifold is similar to that in [15], but our result is essentially different. We treat GD instead of SGD, and in our case, the motion along the manifold is made possible by the subquadratic landscape around the minima, instead of the SGD noise.…”

“…The mechanism of SGD's exploration among different minima is made clear in the recent work [15], which characterizes the movement of SGD iterators along the minima manifold. This picture of exploration along minima manifolds suits the neural network problem better than the exploration among isolated minima.…”

“…This picture of exploration along minima manifolds suits the neural network problem better than the exploration among isolated minima. Prior to [15], similar analysis has been conduct for SGD with label noise [6,2]. In this paper, we also analyze the motion of optimizers along the minima manifold (See Section 4.3).…”

Beyond the Quadratic Approximation: The Multiscale Structure of Neural Network Loss Landscapes

“…The mechanism of SGD's exploration among different minima is made clear in the recent work [14], which characterizes the movement of SGD iterators along the minima manifold. This picture of exploration along minima manifolds suits the neural network problem better than the exploration among isolated minima.…”