Improved Overparametrization Bounds for Global Convergence of Stochastic Gradient Descent for Shallow Neural Networks

Abstract

We study the overparametrization bounds required for the global convergence of stochastic gradient descent algorithm for a class of one hidden layer feed-forward neural networks, considering most of the activation functions used in practice, including ReLU. We improve the existing state-of-the-art results in terms of the required hidden layer width. We introduce a new proof technique combining nonlinear analysis with properties of random initializations of the network. First, we establish the global convergence of continuous solutions of the differential inclusion being a nonsmooth analogue of the gradient flow for the MSE loss. Second, we provide a technical result (working also for general approximators) relating solutions of the aforementioned differential inclusion to the (discrete) stochastic gradient descent sequences, and hence establishing linear convergence towards zero loss for the stochastic gradient descent iterations.

GANs training: A game and stochastic control approach

“…For example, Chizat and Bach (2018) transform the original nonconvex minimization problem to a convex one and shows that the limit points of a suitable gradient algorithm are global minima of the convex problem. In the same spirit, Polaczyk and Cyranka (2022) show that combining over-parametrization and random initialization techniques produces almost convex and smooth objective functions that enable neural networks to avoid local minima. Du et al (2019) and Polaczyk and Cyranka (2022) also rely on over-parametrization to prove the convergence of gradient descent for feedforward and ResNet neural networks.…”

“…In the same spirit, Polaczyk and Cyranka (2022) show that combining over-parametrization and random initialization techniques produces almost convex and smooth objective functions that enable neural networks to avoid local minima. Du et al (2019) and Polaczyk and Cyranka (2022) also rely on over-parametrization to prove the convergence of gradient descent for feedforward and ResNet neural networks. More generally, Ghadimi and Lan (2016) and Ghadimi and Lan (2013) extend the convergence results of gradient-based methods to a class larger than the set of convex objective functions.…”