Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes

Dereich, Steffen; Kassing, Sebastian

doi:10.48550/arxiv.2102.09385

Cited by 11 publications

(13 citation statements)

References 13 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is empirical evidence that these deep learning-based methods work well at least for medium prescribed accuracies; see, e.g., the simulations in [13,18,32,4,3]. However, stochastic optimization methods may get trapped in local minima and there exists no theoretical convergence result; cf., e.g., [17].…”

Section: Introductionmentioning

confidence: 99%

Multilevel Picard Approximations of High-Dimensional Semilinear Parabolic Differential Equations with Gradient-Dependent Nonlinearities

Hutzenthaler¹,

Kruse²

2020

SIAM J. Numer. Anal.

View full text Add to dashboard Cite

Parabolic partial differential equations (PDEs) and backward stochastic differential equations (BSDEs) have a wide range of applications. In particular, high-dimensional PDEs with gradient-dependent nonlinearities appear often in the state-of-the-art pricing and hedging of financial derivatives. In this article we prove that semilinear heat equations with gradient-dependent nonlinearities can be approximated under suitable assumptions with computational complexity that grows polynomially both in the dimension and the reciprocal of the accuracy.Picard approximations (5) with Gauß-Legendre quadrature rules given by (4). The reason for choosing Gauß-Legendre quadrature rules is the very fast convergence in case of sufficiently smooth integrands; cf. Lemma 4.5 below. Fast readers can then jump to Corollary 4.8, which is the main result of this article. For the proof of Corollary 4.8, we first derive the (recursive) bound (54) for the global error and then iterate this inequality to obtain the (non-recursive) bound (65) for the global error. Finally Lemma 3.3 provides an upper bound for the iterated Gauß-Legendre integrals over inverse square roots appearing in (65).

show abstract

Section: Introductionmentioning

confidence: 99%

Multilevel Picard Approximations of High-Dimensional Semilinear Parabolic Differential Equations with Gradient-Dependent Nonlinearities

Hutzenthaler¹,

Kruse²

2020

SIAM J. Numer. Anal.

View full text Add to dashboard Cite

show abstract

“…In this section we establish in Proposition 9.4 below an abstract local convergence result for GD under a Kurdyka-Lojasiewicz assumption. In the scientific literature similar convergence results for GD type processes under a Lojasiewicz assumption can be found, e.g., in Absil et al [1], Attouch & Bolte [3], Dereich & Kassing [21], and Xu & Yin [63]. To prove Proposition 9.4 we transfer the ideas from the continuous-time setting in Section 8 to the discrete-time setting.…”

Section: Convergence Analysis For Gd Processesmentioning

confidence: 63%

“…Regarding abstract results on the convergence of GF and GD processes we refer, for example, to [5,34,49,50,56] for the case of convex objective functions, we refer, for instance, to [1,3,4,10,21,39,42,43,46,47,51] for convergence results for GF and GD processes under Lojasiewicz type conditions, and we refer, for instance, to [7,27,44,54] and the references mentioned therein for further results without convexity conditions. In general, without global assumptions on the objective function such as convexity, gradient-based methods may converge to non-global local minima or saddle points.…”

Section: Introduction and Main Resultsmentioning

confidence: 99%

On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

Jentzen,

Riekert

2021

Preprint

View full text Add to dashboard Cite

Although gradient descent (GD) optimization methods in combination with rectified linear unit (ReLU) artificial neural networks (ANNs) often supply an impressive performance in real world learning problems, till this day it remains -in all practically relevant scenarios -an open problem of research to rigorously prove (or disprove) the conjecture that such GD optimization methods do converge in the training of ANNs with ReLU activation.In this article we study fully-connected feedforward deep ReLU ANNs with an arbitrarily large number of hidden layers and we prove convergence of the risk of the GD optimization method with random initializations in the training of such ANNs under the assumption that the unnormalized probability density function p : [a, b] d → [0, ∞) of the probability distribution of the input data of the considered supervised learning problem is piecewise polynomial, under the assumption that the target function f : [a, b] d → R δ (describing the relationship between input data and the output data) is piecewise polynomial, and under the assumption that the risk function of the considered supervised learning problem admits at least one regular global minimum. In addition, in the special situation of shallow ANNs with just one hidden layer and one-dimensional input we also verify this assumption by proving in the training of such shallow ANNs that for every Lipschitz continuous target function there exists a global minimum in the risk landscape. Finally, in the training of deep ANNs with ReLU activation we also study solutions of gradient flow (GF) differential equations and we prove that every non-divergent GF trajectory converges with a polynomial rate of convergence to a critical point (in the sense of limiting Fréchet subdifferentiability).Our mathematical convergence analysis builds up on tools from real algebraic geometry such as the concept of semi-algebraic functions and generalized Kurdyka-Lojasiewicz inequalities, on tools from functional analysis such as the Arzelà-Ascoli theorem on the relative compactness of uniformly bounded and equicontinuous sequences of continuous functions, on tools from nonsmooth analysis such as the concept of limiting Fréchet subgradients, as well as on the fact that the set of realization functions of shallow ReLU ANNs with fixed architecture forms a closed subset of the set of continuous functions revealed in [P. Petersen, M. Raslan, & F. Voigtlaender, Topological properties of the set of functions generated by neural networks of fixed size.

show abstract

“…There are several promising attempts in the scientific literature which intend to mathematically analyze GD optimization algorithms in the training of ANNs. In particular, there are various convergence results for GD optimization algorithms in the training of ANNs that assume convexity of the considered objective functions (cf., e.g., [4,5,24] and the references mentioned therein), there are general abstract convergence results for GD optimization algorithms that do not assume convexity of the considered objective functions (cf., e.g., [1,6,10,14,19,20,22] and the references mentioned therein), there are divergence results and lower bounds for GD optimization algorithms in the training of ANNs (cf., e.g., [8,18,23] and the references mentioned therein), there are mathematical analyzes regarding the initialization in the training of ANNs with GD optimization algorithms (cf., e.g., [15,16,23,27] and the references mentioned therein), and there are convergence results for GD optimization algorithms in the training of ANNs in the case of constant target functions (cf. [7]).…”

Section: Introductionmentioning

confidence: 99%

Convergence rates for gradient descent in the training of overparameterized artificial neural networks with biases

Jentzen¹,

Kröger²

2021

Preprint

View full text Add to dashboard Cite

In recent years, artificial neural networks have developed into a powerful tool for dealing with a multitude of problems for which classical solution approaches reach their limits. However, it is still unclear why randomly initialized gradient descent optimization algorithms, such as the well-known batch gradient descent, are able to achieve zero training loss in many situations even though the objective function is non-convex and non-smooth. One of the most promising approaches to solving this problem in the field of supervised learning is the analysis of gradient descent optimization in the so-called overparameterized regime. In this article we provide a further contribution to this area of research by considering overparameterized fully-connected rectified artificial neural networks with biases. Specifically, we show that for a fixed number of training data the mean squared error using batch gradient descent optimization applied to such a randomly initialized artificial neural network converges to zero at a linear convergence rate as long as the width of the artificial neural network is large enough, the learning rate is small enough, and the training input data are pairwise linearly independent.

show abstract

Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes

Cited by 11 publications

References 13 publications

Multilevel Picard Approximations of High-Dimensional Semilinear Parabolic Differential Equations with Gradient-Dependent Nonlinearities

Multilevel Picard Approximations of High-Dimensional Semilinear Parabolic Differential Equations with Gradient-Dependent Nonlinearities

On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks

Convergence rates for gradient descent in the training of overparameterized artificial neural networks with biases

Contact Info

Product

Resources

About