Overall error analysis for the training of deep neural networks via stochastic gradient descent with random initialisation

Abstract

In spite of the accomplishments of deep learning based algorithms in numerous applications and very broad corresponding research interest, at the moment there is still no rigorous understanding of the reasons why such algorithms produce useful results in certain situations. A thorough mathematical analysis of deep learning based algorithms seems to be crucial in order to improve our understanding and to make their implementation more effective and efficient. In this article we provide a mathematically rigorous full error analysis of deep learning based empirical risk minimisation with quadratic loss function in the probabilistically strong sense, where the underlying deep neural networks are trained using stochastic gradient descent with random initialisation. The convergence speed we obtain is presumably far from optimal and suffers under the curse of dimensionality. To the best of our knowledge, we establish, however, the first full error analysis in the scientific literature for a deep learning based algorithm in the probabilistically strong sense and, moreover, the first full error analysis in the scientific literature for a deep learning based algorithm where stochastic gradient descent with random initialisation is the employed optimisation method.

A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions

“…This, ( 24), (25), and the fact that for all x ∈ [a, b] d it holds that 0 (x) = 0 prove that for all φ, ψ ∈ K we have that…”

“…For more detailed overviews and further references on SGD optimization schemes we refer, e.g., to [8], [18,Section 1.1], [23, Section 1], and [39]. The effect of random initializations in the training of ANNs was studied, e.g., in [6,20,21,25,32,42] and the references mentioned therein. Another promising branch of research has investigated the convergence of SGD for the training of ANNs in the so-called overparametrized regime, where the number of ANN parameters has to be sufficiently large.…”

On the approximation of functions by tanh neural networks

“…the number of non-zero weights and biases). Moreover, many bounds on the generalization error require an estimate of the network width [3,26].…”

Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes

“…Hence, in that case our results show that a SGD scheme associated with the training of the network converges almost surely on the event of staying local. Concerning the training of neural networks via SGD we refer the reader to [BM11] [JW20]. Related target functions (loss landscapes) are analysed in [Coo18], [Ngu19], [Coo20], [PRV20] and [QZX20].…”