Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints

Abstract

Algorithm-dependent generalization error bounds are central to statistical learning theory. A learning algorithm may use a large hypothesis space, but the limited number of iterations controls its model capacity and generalization error. The impacts of stochastic gradient methods on generalization error for non-convex learning problems not only have important theoretical consequences, but are also critical to generalization errors of deep learning.In this paper, we study the generalization errors of Stochastic Gradient Langevin Dynamics (SGLD) with non-convex objectives. Two theories are proposed with non-asymptotic discretetime analysis, using Stability and PAC-Bayesian results respectively. The stability-based theory obtains a bound of O 1 n L √ βT k , where L is uniform Lipschitz parameter, β is inverse temperature, and T k is aggregated step sizes. For PAC-Bayesian theory, though the bound has a slower O(1/ √ n) rate, the contribution of each step is shown with an exponentially decaying factor by imposing 2 regularization, and the uniform Lipschitz constant is also replaced by actual norms of gradients along trajectory. Our bounds have no implicit dependence on dimensions, norms or other capacity measures of parameter, which elegantly characterizes the phenomenon of "Fast Training Guarantees Generalization" in non-convex settings. This is the first algorithmdependent result with reasonable dependence on aggregated step sizes for non-convex learning, and has important implications to statistical learning aspects of stochastic gradient methods in complicated models such as deep learning.

Stability and Convergence Trade-off of Iterative Optimization Algorithms

“…We study this particular step-size setting of SGLD. It has been shown by Mou et al (2017) that SGLD has the following uniform stability for L-Lipschitz convex loss function,…”

“…It remains unclear, however, what is the algorithmic stability of general iterative optimization algorithms. Recently, to show the effectiveness of commonly used optimization algorithms in many large-scale learning problems, algorithmic stability has been established for stochastic gradient methods (Hardt et al, 2016), stochastic gradient Langevin dynamics (Mou et al, 2017), as well as for any algorithm in situations where global minima are approximately achieved (Charles and Papailiopoulos, 2017).…”

“…The stability concept has an intuitive appeal: an algorithm is stable if it is robust to small perturbations in the composition of the learning data set. Recently it has been shown that algorithmic stability is well suited for controlling generalization error of stochastic gradient methods (Hardt et al, 2016), as well as stochastic gradient Langevin dynamics algorithm (Mou et al, 2017).…”

Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks

“…However, how to extend these results to non-linear neural networks remains unclear (Wei et al, 2018). Another line of algorithm-dependent analysis of generalization (Hardt et al, 2015;Mou et al, 2017;Chen et al, 2018) used stability of specific optimization algorithms that satisfy certain generic properties like convexity, smoothness, etc. However, as the number of epochs becomes large, these generalization bounds are vacuous.…”

Laplacian Smoothing Stochastic Gradient Markov Chain Monte Carlo

“…From the theoretical perspective, the convergence guarantee of SGLD has been proved for both strongly log-concave distributions (Dalalyan and Karagulyan, 2017) and non-log-concave distributions (Raginsky et al, 2017;Xu et al, 2018) in 2-Wasserstein distance. Mou et al (2017) further studied the generalization performance of SGLD for nonconvex optimization. Although SGLD can drastically reduce the computational cost, it is also observed to have a slow convergence rate due to the large variance caused by the stochastic gradient .…”