Hidden unit specialization in layered neural networks: ReLU vs. sigmoidal activation

Oostwal, Elisa; Straat, Michiel; Biehl, Michael

doi:10.1016/j.physa.2020.125517

Cited by 47 publications

(39 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Phase change behavior of dynamical systems using the sigmoid and RELU activation functions are known in the literature in the context of generalization performance of deep neural networks [ 34 , 35 ]. In this section we present a complete proof of the bifurcation analysis of non-linear dynamical systems involving sigmoid activation function despite its connections with [ 34 , 35 ]. Our results in Section 5.1 provide a more complete picture of the behavior of the dynamics in all regimes and can be readily exploited to analyze the dynamics of ( 9 ) and ( 10 ).…”

Section: Resultsmentioning

confidence: 99%

Dynamics of Coordinate Ascent Variational Inference: A Case Study in 2D Ising Models

Plummer

Pati

Bhattacharya

2020

Entropy

View full text Add to dashboard Cite

Variational algorithms have gained prominence over the past two decades as a scalable computational environment for Bayesian inference. In this article, we explore tools from the dynamical systems literature to study the convergence of coordinate ascent algorithms for mean field variational inference. Focusing on the Ising model defined on two nodes, we fully characterize the dynamics of the sequential coordinate ascent algorithm and its parallel version. We observe that in the regime where the objective function is convex, both the algorithms are stable and exhibit convergence to the unique fixed point. Our analyses reveal interesting discordances between these two versions of the algorithm in the region when the objective function is non-convex. In fact, the parallel version exhibits a periodic oscillatory behavior which is absent in the sequential version. Drawing intuition from the Markov chain Monte Carlo literature, we empirically show that a parameter expansion of the Ising model, popularly called the Edward–Sokal coupling, leads to an enlargement of the regime of convergence to the global optima.

show abstract

Section: Resultsmentioning

confidence: 99%

Dynamics of Coordinate Ascent Variational Inference: A Case Study in 2D Ising Models

Plummer

Pati

Bhattacharya

2020

Entropy

View full text Add to dashboard Cite

show abstract

“…The 3000 images of corn kernels were trained in the yolov4-tiny model after preprocessing. In order to simplify the calculation process, Leaky ReLU was used as the activation function of CSPDarknet [26,27]. The expression of Leaky ReLU is shown in Equation (2):…”

Section: Evaluation Of the Yolov4-tiny Modelmentioning

confidence: 99%

Design and Experiment of a Broken Corn Kernel Detection Device Based on the Yolov4-Tiny Algorithm

Yao

et al. 2021

Agriculture

View full text Add to dashboard Cite

At present, the wide application of the CNN (convolutional neural network) algorithm has greatly improved the intelligence level of agricultural machinery. Accurate and real-time detection for outdoor conditions is necessary for realizing intelligence and automation of corn harvesting. In view of the problems with existing detection methods for judging the integrity of corn kernels, such as low accuracy, poor reliability, and difficulty in adapting to the complicated and changeable harvesting environment, this paper investigates a broken corn kernel detection device for combine harvesters by using the yolov4-tiny model. Hardware construction is first designed to acquire continuous images and processing of corn kernels without overlap. Based on the images collected, the yolov4-tiny model is then utilized for training recognition of the intact and broken corn kernels samples. Next, a broken corn kernel detection algorithm is developed. Finally, the experiments are carried out to verify the effectiveness of the broken corn kernel detection device. The laboratory results show that the accuracy of the yolov4-tiny model is 93.5% for intact kernels and 93.0% for broken kernels, and the value of precision, recall, and F1 score are 92.8%, 93.5%, and 93.11%, respectively. The field experiment results show that the broken kernel rate obtained by the designed detection device are in good agreement with that obtained by the manually calculated statistic, with differentials at only 0.8%. This study provides a technical reference of a real-time method for detecting a broken corn kernel rate.

show abstract

“…The ST framework offers a clean venue for analyzing optimization-related aspects of neural network models in the spirit of physical reductionism. The success of DL models in the past decade has reinitiated a surge of interest in this framework, e.g., [6,27,40,45]. However, despite the long tradition in the statistical physics community and the wide effort put nowadays by the machine learning community, the perplexing geometry of problem (2) still seems to be out of reach of existing analytic tools in regimes encountered in practice.…”

mentioning

confidence: 99%

“…• We show that symmetry breaking makes it possible to derive analytic expressions for families of spurious minima in the form of fractional power series in terms of d and k. Crucially, in contrast to existing approaches which employ various limiting processes, e.g., [6,27,40,45,28,41,12,32,13], our method operates in the natural regime where d and k are finite.…”

mentioning

confidence: 99%

Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II

Arjevani¹,

Field²

2021

Preprint

View full text Add to dashboard Cite

We study the optimization problem associated with fitting two-layer ReLU neural networks with respect to the squared loss, where labels are generated by a target network. We make use of the rich symmetry structure to develop a novel set of tools for studying families of spurious minima. In contrast to existing approaches which operate in limiting regimes, our technique directly addresses the nonconvex loss landscape for a finite number of inputs d and neurons k, and provides analytic, rather than heuristic, information. In particular, we derive analytic estimates for the loss at different minima, and prove that modulo O(d −1/2 )-terms the Hessian spectrum concentrates near small positive constants, with the exception of Θ(d) eigenvalues which grow linearly with d. We further show that the Hessian spectrum at global and spurious minima coincide to O(d −1/2 )-order, thus challenging our ability to argue about statistical generalization through local curvature. Lastly, our technique provides the exact fractional dimensionality at which families of critical points turn from saddles into spurious minima. This makes possible the study of the creation and the annihilation of spurious minima using powerful tools from equivariant bifurcation theory.

show abstract

Hidden unit specialization in layered neural networks: ReLU vs. sigmoidal activation

Cited by 47 publications

References 29 publications

Dynamics of Coordinate Ascent Variational Inference: A Case Study in 2D Ising Models

Dynamics of Coordinate Ascent Variational Inference: A Case Study in 2D Ising Models

Design and Experiment of a Broken Corn Kernel Detection Device Based on the Yolov4-Tiny Algorithm

Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II

Contact Info

Product

Resources

About