Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation

Abstract

Gradient descent (GD) type optimization schemes are the standard methods to train artificial neural networks (ANNs) with rectified linear unit (ReLU) activation. Such schemes can be considered as discretizations of gradient flows (GFs) associated to the training of ANNs with ReLU activation and most of the key difficulties in the mathematical convergence analysis of GD type optimization schemes in the training of ANNs with ReLU activation seem to be already present in the dynamics of the corresponding GF differential equations. It is the key subject of this work to analyze such GF differential equations in the training of ANNs with ReLU activation and three layers (one input layer, one hidden layer, and one output layer). In particular, in this article we prove in the case where the target function is possibly multi-dimensional and continuous and in the case where the probability distribution of the input data is absolutely continuous with respect to the Lebesgue measure that the risk of every bounded GF trajectory converges to the risk of a critical point. In addition, in this article we show in the case of a 1-dimensional affine linear target function and in the case where the probability distribution of the input data coincides with the standard uniform distribution that the risk of every bounded GF trajectory converges to zero if the initial risk is sufficiently small. Finally, in the special situation where there is only one neuron on the hidden layer (1-dimensional hidden layer) we strengthen the above named result for affine linear target functions by proving that that the risk of every (not necessarily bounded) GF trajectory converges to zero if the initial risk is sufficiently small.

Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation

“…As in [25, Setting 2.1 and Proposition 2.3] (cf. also [10,24]), we accomplish this, first, by approximating the ReLU activation function through continuously differentiable functions which converge pointwise to the ReLU activation function and whose derivatives converge pointwise to the left derivative of the ReLU activation function and, thereafter, by specifying the generalized gradient function as the limit of the gradients of the approximated risk functions; see (1.1) and (1.3) in Theorem 1.1 and (1.7) and (1.8) in Theorem 1.2 for details.…”

On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation

“…, of zeros of the generalized gradient function G : R 4 → R 4 . In Corollary 5.9 in Subsection 5.6 we extend Corollary 5.8 by using [19,Item (v) in Theorem 1.1] and [11,Theorem 1.2] to establish that in the training of such ANNs we have that the risk of every non-divergent GF trajectory converges to the risk of a global minimum point provided that the initial risk is sufficiently small. The remainder of this section is organized in the following way.…”

“…Finally, in Section 5 we prove in Corollary 5.8 in the special situation where the target function is continuous and piecewise polynomial and where both the input layer and hidden layer of the considered ANNs are one-dimensional that there exist only finitely many different realization functions of all criticial points of the risk function (of all zeros of the generalized gradient function). Theorem 1.2 above can then be shown by combining Proposition 2.12, Corollary 5.8, [19,Item (v) in Theorem 1.1], and [11,Theorem 1.2]. This is precisely the subject of Corollary 5.9 in Section 5.…”

“…This enables us to conclude in item (ii) in Theorem 1.2 that (in contrast to the situation of Theorem 1.1 above) there exist at most finitely many different realization functions of (non-global) local minimum points. In addition, item (i) in Theorem 1.2 together with [19,Item (v) in Theorem 1.1] and [11,Theorem 1.2] allows us to conclude in item (iii) in Theorem 1.2 that in training of such ANNs we have that the risk of every non-divergent GF trajectory converges to the risk of a global minimum point provided that the initial risk is sufficiently small. To describe a GF trajectory, we need to specify an appropriate generalized gradient function in Theorem 1.2 as the risk function is not differentiable in the case of ANNs with ReLU activation (due to the fact that the ReLU activation function R x → max{x, 0} ∈ R fails to be differentiable in the origin).…”

“…Note that [11, Theorem 1.2] ensures that sup t∈[0,∞) Θ t < ∞. The fact that for all θ ∈ G −1 ({0}) ∩ (R ∞ ) −1 ((inf ϑ∈R 4 R ∞ (ϑ), ∞)) it holds that R ∞ (Θ 0 ) ≤ ε + inf ϑ∈R 4 R ∞ (ϑ) < R ∞ (θ) (5.63)and[19, Item (v) in Theorem 1.1] therefore show that lim sup t→∞ R ∞ (Θ t ) = inf ϑ∈R 4 R ∞ (ϑ). (5.64)…”