Embedding Principle: a hierarchical structure of loss landscape of deep neural networks

Zhang, Yaoyu; Li, Yuqing; Zhang, Zhongwang; Luo, Tao; Xu, Zhi‐Qin John

doi:10.48550/arxiv.2111.15527

Cited by 3 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In general, without global assumptions on the objective function such as convexity, gradient-based methods may converge to non-global local minima or saddle points. It therefore becomes important to analyze critical points of the objective function in the training of ANNs and we refer, for example, to [14,65,68,73,74] for articles which study the appearance of critical points in the risk landscape in the training of ANNs. The question under which conditions gradientbased optimization algorithms cannot converge to saddle points was investigated, for example, in [32,48,49,58,59].…”

Section: Introduction and Main Resultsmentioning

confidence: 99%

On the Existence of Global Minima and Convergence Analyses for Gradient Descent Methods in the Training of Deep Neural Networks

Jentzen¹,

Riekert²

2022

JML

View full text Add to dashboard Cite

Section: Introduction and Main Resultsmentioning

confidence: 99%

On the Existence of Global Minima and Convergence Analyses for Gradient Descent Methods in the Training of Deep Neural Networks

Jentzen¹,

Riekert²

2022

JML

View full text Add to dashboard Cite

“…In general, without global assumptions on the objective function such as convexity, gradient-based methods may converge to non-global local minima or saddle points. It therefore becomes important to analyze critical points of the objective function in the training of ANNs and we refer, for example, to [14,59,61,65,66] for articles which study the appearance of critical points in the risk landscape in the training of ANNs. The question under which conditions gradient-based optimization algorithms cannot converge to saddle points was investigated, for example, in [28,42,43,52,53].…”

Section: Introduction and Main Resultsmentioning

confidence: 99%

“… 65). Furthermore, observe that (3.62) and(3.64) prove that for all k ∈ N it holds that Z 2 (k) ∈ (Df )(Z 1 (k)).…”

mentioning

confidence: 92%

On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

Jentzen,

Riekert

2021

Preprint

View full text Add to dashboard Cite

Although gradient descent (GD) optimization methods in combination with rectified linear unit (ReLU) artificial neural networks (ANNs) often supply an impressive performance in real world learning problems, till this day it remains -in all practically relevant scenarios -an open problem of research to rigorously prove (or disprove) the conjecture that such GD optimization methods do converge in the training of ANNs with ReLU activation.In this article we study fully-connected feedforward deep ReLU ANNs with an arbitrarily large number of hidden layers and we prove convergence of the risk of the GD optimization method with random initializations in the training of such ANNs under the assumption that the unnormalized probability density function p : [a, b] d → [0, ∞) of the probability distribution of the input data of the considered supervised learning problem is piecewise polynomial, under the assumption that the target function f : [a, b] d → R δ (describing the relationship between input data and the output data) is piecewise polynomial, and under the assumption that the risk function of the considered supervised learning problem admits at least one regular global minimum. In addition, in the special situation of shallow ANNs with just one hidden layer and one-dimensional input we also verify this assumption by proving in the training of such shallow ANNs that for every Lipschitz continuous target function there exists a global minimum in the risk landscape. Finally, in the training of deep ANNs with ReLU activation we also study solutions of gradient flow (GF) differential equations and we prove that every non-divergent GF trajectory converges with a polynomial rate of convergence to a critical point (in the sense of limiting Fréchet subdifferentiability).Our mathematical convergence analysis builds up on tools from real algebraic geometry such as the concept of semi-algebraic functions and generalized Kurdyka-Lojasiewicz inequalities, on tools from functional analysis such as the Arzelà-Ascoli theorem on the relative compactness of uniformly bounded and equicontinuous sequences of continuous functions, on tools from nonsmooth analysis such as the concept of limiting Fréchet subgradients, as well as on the fact that the set of realization functions of shallow ReLU ANNs with fixed architecture forms a closed subset of the set of continuous functions revealed in [P. Petersen, M. Raslan, & F. Voigtlaender, Topological properties of the set of functions generated by neural networks of fixed size.

show abstract

“…Kawaguchi [22] and Laurent & von Brecht [24]). For a connection between the critical points of the risk function and the critical points of the risk function with regard to a larger network width we refer to the articles Zhang et al [36,37].…”

Section: Introductionmentioning

confidence: 99%

On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation

Ibragimov¹,

Jentzen²,

Kröger³

et al. 2022

Preprint

View full text Add to dashboard Cite

Gradient descent (GD) type optimization schemes are the standard instruments to train fully connected feedforward artificial neural networks (ANNs) with rectified linear unit (ReLU) activation and can be considered as temporal discretizations of solutions of gradient flow (GF) differential equations. It has recently been proved that the risk of every bounded GF trajectory converges in the training of ANNs with one hidden layer and ReLU activation to the risk of a critical point, by which we mean a zero point of the corresponding gradient function. Taking this into account it is one of the key research issues in the mathematical convergence analysis of GF trajectories and GD type optimization schemes, respectively, to study sufficient and necessary conditions for critical points of the risk function and, thereby, to obtain an understanding about the appearance of critical points in dependence of the problem parameters such as the target function. In the first main result of this work we prove in the training of ANNs with one hidden layer and ReLU activation that for every ∈ R, ∈ ( , ∞) and every arbitrarily large positive δ ∈ (0, ∞) we have that there exists a Lipschitz continuous target function : [ , ] → R such that for every number H ∈ N ∩ (1, ∞) of neurons on the hidden layer we have that the risk function has uncountably many different realization functions of non-global local minimum points whose risks are strictly larger than the sum of the risk of the global minimum points and the arbitrarily large positive real number δ. In the second main result of this work we show in the training of ANNs with one hidden layer and ReLU activation in the special situation where there is only one neuron on the hidden layer and where the target function is continuous and piecewise polynomial that there exist at most finitely many different realization functions of critical points.

show abstract

Embedding Principle: a hierarchical structure of loss landscape of deep neural networks

Cited by 3 publications

References 25 publications

On the Existence of Global Minima and Convergence Analyses for Gradient Descent Methods in the Training of Deep Neural Networks

On the Existence of Global Minima and Convergence Analyses for Gradient Descent Methods in the Training of Deep Neural Networks

On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

On the existence of infinitely many realization functions of non-global local minima in the training of artificial neural networks with ReLU activation

Contact Info

Product

Resources

About