Stochastic Gradient Descent for Nonconvex Learning Without Bounded Gradient Assumptions

Abstract

Stochastic gradient descent (SGD) is a popular and efficient method with wide applications in training deep neural nets and other nonconvex models. While the behavior of SGD is well understood in the convex learning setting, the existing theoretical results for SGD applied to nonconvex objective functions are far from mature. For example, existing results require to impose a nontrivial assumption on the uniform boundedness of gradients for all iterates encountered in the learning process, which is hard to verify in practical implementations. In this paper, we establish a rigorous theoretical foundation for SGD in nonconvex learning by showing that this boundedness assumption can be removed without affecting convergence rates. In particular, we establish sufficient conditions for almost sure convergence as well as optimal convergence rates for SGD applied to both general nonconvex objective functions and gradient-dominated objective functions. A linear convergence is further derived in the case with zero variances.

Stochastic Gradient Descent on Nonconvex Functions with General Noise Models

“…Our goal here is to move towards a more general theory of convergence that combines all of these threads under a single analysis framework. Specifically, by innovating on the strategies of Lei et al [2019] and Patel [2020], we will prove the following results for SGD with matrix-valued learning rates, which we state informally now and formalize later.…”

“…with α ∈ (0, 1], the objective function, F , evaluated at the SGD iterates converges almost surely to a bounded random variable. Moreover, Lei et al [2019] show that, when α = 1, the expected value of the norm of the gradient function, Ḟ , evaluated at the SGD iterates converges to zero. 3.…”

“…1 Owing to its broad usage, SGD's global behavior on different classes of functions f (and, hence, F ) has been of substantial interest. While there are many works that have provided insight, understanding SGD's global behavior has been notably advanced by several recent works [Asi and Duchi, 2019, Lei et al, 2019, Patel, 2020, Khaled and Richtárik, 2020 that we overview presently.…”

Neural network regression for Bermudan option pricing

“…It is usually solved in practice using toolboxes as Scikit-Learn or TensorFlow, by means of a stochastic gradient descent method for which a full convergence proof under realistic assumptions are still unknown in our knowledge. See Bottou et al [2018] or E et al [2020] for recent in depth reviews of these subjects and Ghadimi and Lan [2013], Lei et al [2019], Fehrman et al [2020] for results for a non convex function.…”

Sparse Non-Convex Optimization For Higher Moment Portfolio Management

“…There are four major approaches to handles nonconvex optimization problems. The first groups of methods is based on stochastic gradient descent and its convergence rates are far from other methods as is described in (Lei et al 2020). The second approach is to see noncovex functions as a difference of some convex functions but this framework does not work for any class of nonconvex problems.…”