Spectral methods to study the robustness of residual neural networks with infinite layers

Trimborn, Torsten; Gerster, Stephan; Visconti, Giuseppe

doi:10.3934/fods.2020012

Cited by 4 publications

(6 citation statements)

References 63 publications

(73 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…matrix K ∈ ℝ m×m , a factor 𝜈 > 0 , and a m−dimensional standard Wiener process dW, the corresponding dynamics reads Mathematically, the feature of presenting a possibly large number N of input and target data to the ANN can be exploited in terms of the mean-field limit [21,[30][31][32][33]. Instead of considering input data (0) i individually, we consider a statistical description of these data.…”

Section: Mathematical Formulation Of Residual Neural Network (Resnets)mentioning

confidence: 99%

See 1 more Smart Citation

Machine learning tools in production engineering

Rom

Brockmann

Herty

et al. 2022

Int J Adv Manuf Technol

View full text Add to dashboard Cite

Machine learning methods have shown potential for the optimization of production processes. Due to the complex relationships often inherent in those processes, the success of such methods is often uncertain and unreliable. Therefore, understanding the (algorithmic) behavior and results of machine learning methods is crucial to improve the prediction of production processes. Here, mathematical tools may help. This paper shows how efficient algorithms for the training of neural networks and their retraining in the framework of transfer learning are expressed in a discrete as well as a time-continuous formulation. The latter can be analyzed and investigated using mathematical techniques from kinetic gas dynamics. The results obtained provide a first step towards explainable artificial intelligence. Based on the mathematical description, an adapted ensemble method for retraining of neural networks is proposed and compared with backpropagation algorithms. The process of training and retraining is a common task and therefore demonstrated for two very different production processes. The first one involves the prediction of specific cutting forces and the second one the prediction of particle properties in a plasma spraying coating process. For both use cases, the presented algorithms are applied and their performance is evaluated giving thereby an indication how mathematically inspired methods improve classical tasks in production processes.

show abstract

Section: Mathematical Formulation Of Residual Neural Network (Resnets)mentioning

confidence: 99%

“…In [15], the time-continuous version of a ResNet is studied and different temporal discretization schemes are discussed. There are also studies on application of kinetic methods to ResNets [18][19][20][21]. For example, in [20], the authors consider the limit of infinitely many neurons and gradient steps in the case of one hidden layer.…”

Section: Introductionmentioning

confidence: 99%

Machine learning tools in production engineering

Rom

Brockmann

Herty

et al. 2022

Int J Adv Manuf Technol

View full text Add to dashboard Cite

show abstract

“…Since the two PDEs are decoupled, they can be solved simultaneously. Application of the method of lines to (22) on discrete cells Ω j , defining a discretization of the physical domain Ω, leads to the coupled system of ODEs…”

Section: Numerical Discretization Schemementioning

confidence: 99%

“…However, neither the training nor the well-posedness have been analyzed so far. Using a mean-field or kinetic description of large-scale neural networks has so far only be discussed for particular examples in a few recent manuscripts [17][18][19][20][21][22]. A general investigation in particular in view of large input data is to the best of our knowledge still open.…”

Section: Introductionmentioning

confidence: 99%

Continuous limits of residual neural networks in case of large input data

Herty

Thünen

Trimborn³

et al. 2022

Communications in Applied and Industrial Mathematics

Self Cite

View full text Add to dashboard Cite

Residual deep neural networks (ResNets) are mathematically described as interacting particle systems. In the case of infinitely many layers the ResNet leads to a system of coupled system of ordinary differential equations known as neural differential equations. For large scale input data we derive a mean–field limit and show well–posedness of the resulting description. Further, we analyze the existence of solutions to the training process by using both a controllability and an optimal control point of view. Numerical investigations based on the solution of a formal optimality system illustrate the theoretical findings.

show abstract

“…However, neither the training nor the well-posedness have been analyzed so far. Using a mean-field or kinetic description of large-scale neural networks has so far only be discussed for particular examples in a few recent manuscripts [2,8,29,36,37,39]. A general investigation in particular in view of large input data is to the best of our knowledge still open.…”

Section: Introductionmentioning

confidence: 99%

Continuous limits of residual neural networks in case of large input data

Herty¹,

Thuenen²,

Trimborn³

et al. 2021

Preprint

View full text Add to dashboard Cite

Residual deep neural networks (ResNets) are mathematically described as interacting particle systems. In the case of infinitely many layers the ResNet leads to a system of coupled system of ordinary differential equations known as neural differential equations. For large scale input data we derive a mean-field limit and show well-posedness of the resulting description. Further, we analyze the existence of solutions to the training process by using both a controllability and an optimal control point of view. Numerical investigations based on the solution of a formal optimality system illustrate the theoretical findings.

show abstract

Spectral methods to study the robustness of residual neural networks with infinite layers

Cited by 4 publications

References 63 publications

Machine learning tools in production engineering

Machine learning tools in production engineering

Continuous limits of residual neural networks in case of large input data

Continuous limits of residual neural networks in case of large input data

Contact Info

Product

Resources

About