Learning dynamics on different timescales

Endres, Dominik; Riegler, Peter

doi:10.1088/0305-4470/32/49/306

Cited by 4 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[46]- [48]. Decoupling the selection of these few additional parameters from the training of the input to hidden weights should be possible following the ideas presented in [49].…”

Section: Discussionmentioning

confidence: 99%

Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation

Oostwal,

Straat,

Biehl

2019

Preprint

View full text Add to dashboard Cite

We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is in the center of interest. We compute typical learning curves for shallow networks with K hidden units in matching student teacher scenarios. The systems exhibit sudden changes of the generalization performance via the process of hidden unit specialization at critical sizes of the training set. Surprisingly, our results show that the training behavior of ReLU networks is qualitatively different from that of networks with sigmoidal activations. In networks with K ≥ 3 sigmoidal hidden units, the transition is discontinuous: Specialized network configurations co-exist and compete with states of poor performance even for very large training sets. On the contrary, the use of ReLU activations results in continuous transitions for all K. For large enough training sets, two competing, differently specialized states display similar generalization abilities, which coincide exactly for large networks in the limit K → ∞. * (who contributed equally)This work was supported through the Northern Netherlands Region of Smart Factories (RoSF) consortium (see: http://www.rosf.nl).E. Oostwal, M. Straat, and M. Biehl are with the Bernoulli Institute for Mathematics,

show abstract

“…[46]- [48]. Decoupling the selection of these few additional parameters from the training of the input to hidden weights should be possible following the ideas presented in [49].…”

Section: Discussionmentioning

confidence: 99%

Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation

Oostwal,

Straat,

Biehl

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The equilibrium values for w i are thus obtained as functions of R ik and Q ij and can be further used to eliminate any appearance of w i in (30,31). See [21] for details.…”

Section: XImentioning

confidence: 99%

Statistical Mechanics of On-line Learning

Biehl

Caticha

Riegler

2009

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. We introduce and discuss the application of statistical physics concepts in the context of on-line machine learning processes. The consideration of typical properties of very large systems allows to perfom averages over the randomness contained in the sequence of training data. It yields an exact mathematical description of the training dynamics in model scenarios. We present the basic concepts and results of the approach in terms of several examples, including the learning of linear separable rules, the training of multilayer neural networks, and Learning Vector Quantization.

show abstract

“…(Cybenko 1989, Hornik 1991, Hanin 2017. Decoupling the selection of these few additional parameters from the training of the input to hidden weights should be possible following the ideas presented in (Endres and Riegler 1999).…”

Section: Discussionmentioning

confidence: 99%

Machine learning: statistical physics based theory and smart industry applications

Straat

View full text Add to dashboard Cite

The contributions in this thesis are divided into two main parts: 1) a theoretical analysis of learning in neural networks and Learning Vector Quantization (LVQ) in model situations using statistical physics techniques and 2) the application of machine learning to smart industry settings.In the first part we address highly relevant situations and questions for current machine learning practice: using tools from statistical physics we analyse the learning behaviour in Rectified Linear Unit (ReLU) neural networks and compare it to sigmoidal neural networks in both on-line and off-line supervised learning settings, in order to contribute to the much needed theoretical insight into the properties of the use of the ReLU function, the most popular type of activation function in deep neural networks that are used in many machine learning tasks. Secondly, we analyse neural networks and LVQ under real and virtual concept drift processes that affect many applications of machine learning systems. Our analyses reveal several significant effects, which are, among others: ReLU networks handle overparameterization differently than sigmoidal networks, ReLU networks exhibit favourable second order phase transitions towards hidden unit specialization instead of the first order phase transitions observed for sigmoidal networks and applying weight decay in concept drift scenarios is more effective for ReLU neural networks, in which it significantly accelerates the onset of specialization. For LVQ we find non-trivial dependence of the generalization performance on the learning rate in concept drift situations. Moreover, it is shown that an appropriate amount of weight decay can be beneficial to the performance in the real drift settings. In contrast, the resulting limited flexibility of the prototypes decreases the performance under virtual drift.In the second part of the thesis we focus on the use of computational intelligence approaches in applications in industry. First, we perform a typical Industry 4.0 case study in collaboration with industry that concerns the development of real-time material quality control in a highthroughput production line of steel-based products. In this case study, material measurements iii taken with a fast non-invasive sensor are related to material properties measured by tensile tests. Due to significant correlations between the two types of measurements, we successfully fit and evaluate a model that can estimate material properties in real-time. Furthermore, it is shown on 108 kilometres of processed steel that the model is able to prevent expensive production problems and that it can indicate a risk of the occurrence of production faults.Additionally, we propose a methodology for time series classification that combines Generalized Matrix Learning Vector Quantization (GMLVQ) formulated for complex-valued data with complex-valued Fourier and wavelet features. On several benchmark datasets and a heart beat classification task, learning in the space of complex-valued coefficients is found to yield bett...

show abstract

Learning dynamics on different timescales

Cited by 4 publications

References 12 publications

Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation

Hidden Unit Specialization in Layered Neural Networks: ReLU vs. Sigmoidal Activation

Statistical Mechanics of On-line Learning

Machine learning: statistical physics based theory and smart industry applications

Contact Info

Product

Resources

About