The loss surfaces of neural networks with general activation functions

Abstract

The loss surfaces of deep neural networks have been the subject of several studies, theoretical and experimental, over the last few years. One strand of work considers the complexity, in the sense of local optima, of high dimensional random functions with the aim of informing how local optimisation methods may perform in such complicated settings. Prior work of Choromanska et al (2015) established a direct link between the training loss surfaces of deep multi-layer perceptron networks and spherical multi-spin glass models under some very strong assumptions on the network and its data. In this work, we test the validity of this approach by removing the undesirable restriction to ReLU activation functions. In doing so, we chart a new path through the spin glass complexity calculations using supersymmetric methods in random matrix theory which may prove useful in other contexts. Our results shed new light on both the strengths and the weaknesses of spin glass models in this context.

A Spin Glass Model for the Loss Surfaces of Generative Adversarial Networks

“…However, the Hessian index constraints are formulated in the natural Hessian matrix ( 16 ), but our spectral calculations proceed from the rewritten form ( 21 ). We find however that we can indeed proceed much as in [ 13 ]. Recall the key Hessian matrix given in ( 16 ) by where , , G is Ginibre, and all are independent.…”

“…We use multi-spin glasses in high dimensions as a toy model for neural network loss surfaces without any further justification, beyond that found in [ 1 , 13 ]. GANs are composed of two networks: generator ( G ) and discriminator ( D ).…”

“…To calculate the complexities, we follow the well-trodden route of Kac-Rice formulae as pioneered by [ 35 , 36 ]. For a fully rigorous treatment, we proceed as in [ 2 , 13 ].…”

“…Examples include: the modified spin glass model of [ 10 ] with some explicitly added ‘signal’; the simple explicitly non-linear model of [ 11 ]; the spiked tensor ‘signal-in-noise’ model of [ 12 ]. In a slightly different direction, [ 13 ] removed one of the main assumptions from the [ 1 ] derivation, and in so doing arrived at a deformed spin-glass model. All of this recent activity sits in the context of earlier work connecting spin-glass objects with simple neural networks [ 14 – 16 ] and, more generally, with image reconstruction and other signal processing problems [ 17 ].…”

“…By employing standard Kac-Rice complexity calculations [ 2 , 35 , 36 ] we are able to reduce the loss landscape complexity calculation to a random matrix theoretic calculation. We then employ various Random Matrix Theory techniques as in [ 13 ] to obtain rigorous, explicit leading order asymptotic results. Our calculations rely on the supersymmetric method in Random Matrix Theory, in particular the approach to calculating limiting spectral densities follows [ 37 ] and the calculation also follows [ 38 , 39 ] in important ways.…”