Equivariant bifurcation, quadratic equivariants, and symmetry breaking for the standard representation of S _k

Abstract

Motivated by questions originating from the study of a class of shallow student-teacher neural networks, methods are developed for the analysis of spurious minima in classes of gradient equivariant dynamics related to neural networks. In the symmetric case, methods depend on the generic equivariant bifurcation theory of irreducible representations of the symmetric group on k symbols, S k ; in particular, the standard representation of S k . It is shown that spurious minima (non-global local minima) do not arise from spontaneous symmetry breaking but rather through a complex deformation of the landscape geometry that can be encoded by a generic S k -equivariant bifurcation. We describe minimal models for forced symmetry breaking that give a lower bound on the dynamic complexity involved in the creation of spurious minima when there is no symmetry. Results on generic bifurcation when there are quadratic equivariants are also proved; this work extends and clarifies results of Ihrig & Golubitsky and Chossat, Lauterbach & Melbourne on the instability of solutions when there are quadratic equivariants.

Symmetry & critical points for a model shallow neural network

“…As mentioned, in [9], it was shown that, empirically, the isotropy groups of spurious minima are large subgroups of S d × S d and so are symmetry breaking. Below, we outline the framework of symmetry breaking [17,11,27,28,16,29] allowing the analytic results given for the associated non-convex optimization landscape. The methods are first illustrated with reference to the simplest S d × S d -isotropy case, assuming the Frobenius norm, and are later described for general isotropy groups and tensor norms.…”

“…In [16], it is proved that adding neurons can turn symmetric spurious minima into saddles; minima of lesser symmetry require more neurons. In [29], generic S d -equivariant steady-state bifurcation is studied, emphasizing the complex geometry of the exterior square and the standard representations along which spurious minima studied in [16] are created and annihilated. 1.3.…”