A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions

Jentzen, Arnulf; Riekert, Adrian

doi:10.48550/arxiv.2104.00277

Cited by 8 publications

(17 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that ( 18) and the assumption that v ∈ R d+1 \{0} imply that v d+1 = 0. Moreover, observe that (18) shows that for all u = (u 1 , . .…”

Section: Continuous Dependence Of Active Neuron Regions On Ann Parame...mentioning

confidence: 99%

“…and [18,Proposition 2.3]) establishes items (i), (ii), (iii), and (iv). The proof of Proposition 2.2 is thus complete.…”

Section: Mathematical Description Of Artificial Neural Network (Annsmentioning

confidence: 99%

“…The behavior of the realization functions of ANNs with one hidden layer and ReLU activation under the GF dynamics has been investigated in more detail in [19,23]. Another recent idea is to consider only very special target functions and we refer, in particular, to [6,18] for convergence results for GF and GD processes in the case of constant target functions. In the more general case of affine linear target functions, the critical points of the risk function were characterized in [7] and parts of the analysis in this article exploit this characterization.…”

Section: Introductionmentioning

confidence: 99%

“…A common procedure to overcome this issue (cf. [18] and [6]) is to approximate the ReLU activation function R ∋ x → max{x, 0} ∈ R through appropriate continuously differentiable functions which converge pointwise to the ReLU activation function and whose derivatives converge pointwise to the left derivative of the ReLU activation function. In Theorem 1.1 the function R ∞ : R → R specifies the ReLU activation function and the functions R r : R → R, r ∈ N, serve as such continuously differentiable approximations of the ReLU activation function; see (1) in Theorem 1.1.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation

Jentzen,

Riekert

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Gradient descent (GD) type optimization schemes are the standard methods to train artificial neural networks (ANNs) with rectified linear unit (ReLU) activation. Such schemes can be considered as discretizations of gradient flows (GFs) associated to the training of ANNs with ReLU activation and most of the key difficulties in the mathematical convergence analysis of GD type optimization schemes in the training of ANNs with ReLU activation seem to be already present in the dynamics of the corresponding GF differential equations. It is the key subject of this work to analyze such GF differential equations in the training of ANNs with ReLU activation and three layers (one input layer, one hidden layer, and one output layer). In particular, in this article we prove in the case where the target function is possibly multi-dimensional and continuous and in the case where the probability distribution of the input data is absolutely continuous with respect to the Lebesgue measure that the risk of every bounded GF trajectory converges to the risk of a critical point. In addition, in this article we show in the case of a 1-dimensional affine linear target function and in the case where the probability distribution of the input data coincides with the standard uniform distribution that the risk of every bounded GF trajectory converges to zero if the initial risk is sufficiently small. Finally, in the special situation where there is only one neuron on the hidden layer (1-dimensional hidden layer) we strengthen the above named result for affine linear target functions by proving that that the risk of every (not necessarily bounded) GF trajectory converges to zero if the initial risk is sufficiently small.

show abstract

“…Note that ( 18) and the assumption that v ∈ R d+1 \{0} imply that v d+1 = 0. Moreover, observe that (18) shows that for all u = (u 1 , . .…”

Section: Continuous Dependence Of Active Neuron Regions On Ann Parame...mentioning

confidence: 99%

“…and [18,Proposition 2.3]) establishes items (i), (ii), (iii), and (iv). The proof of Proposition 2.2 is thus complete.…”

Section: Mathematical Description Of Artificial Neural Network (Annsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation

Jentzen,

Riekert

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Moreover, non-global local minimum points could be found in the risk landscape of ANNs with one hidden layer and ReLU activation in special student-teacher setups with the probability distribution of the input data given by the normal distribution (see Safran & Shamir [31]). In other cases, where the target function has a very simple form, the critical points of the risk landscape are fully characterized and thus all local minimum points are known (see Cheridito et al [2,Corollary 2.15], Cheridito et al [3], and Jentzen & Riekert [17,Corollary 2.11]). Additionally, in the case of ANNs with linear activation and finitely many training data it was shown that all local minimum points of the risk function corresponding to the squared error loss are global minimum points (cf.…”

Section: Introductionmentioning

confidence: 99%

On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation

Ibragimov¹,

Jentzen²,

Kröger³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Gradient descent (GD) type optimization schemes are the standard instruments to train fully connected feedforward artificial neural networks (ANNs) with rectified linear unit (ReLU) activation and can be considered as temporal discretizations of solutions of gradient flow (GF) differential equations. It has recently been proved that the risk of every bounded GF trajectory converges in the training of ANNs with one hidden layer and ReLU activation to the risk of a critical point, by which we mean a zero point of the corresponding gradient function. Taking this into account it is one of the key research issues in the mathematical convergence analysis of GF trajectories and GD type optimization schemes, respectively, to study sufficient and necessary conditions for critical points of the risk function and, thereby, to obtain an understanding about the appearance of critical points in dependence of the problem parameters such as the target function. In the first main result of this work we prove in the training of ANNs with one hidden layer and ReLU activation that for every ∈ R, ∈ ( , ∞) and every arbitrarily large positive δ ∈ (0, ∞) we have that there exists a Lipschitz continuous target function : [ , ] → R such that for every number H ∈ N ∩ (1, ∞) of neurons on the hidden layer we have that the risk function has uncountably many different realization functions of non-global local minimum points whose risks are strictly larger than the sum of the risk of the global minimum points and the arbitrarily large positive real number δ. In the second main result of this work we show in the training of ANNs with one hidden layer and ReLU activation in the special situation where there is only one neuron on the hidden layer and where the target function is continuous and piecewise polynomial that there exist at most finitely many different realization functions of critical points.

show abstract

Data driven control based on Deep Q-Network algorithm for heading control and path following of a ship in calm water and waves

2022

View full text Add to dashboard Cite

A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions

Cited by 8 publications

References 24 publications

Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation

Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation

On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation

Data driven control based on Deep Q-Network algorithm for heading control and path following of a ship in calm water and waves

Contact Info

Product

Resources

About