Nonperturbative renormalization for the neural network-QFT correspondence

Abstract

In a recent work [1], Halverson, Maiti and Stoner proposed a description of neural networks in terms of a Wilsonian effective field theory. The infinite-width limit is mapped to a free field theory while finite N corrections are taken into account by interactions (non-Gaussian terms in the action). In this paper, we study two related aspects of this correspondence. First, we comment on the concepts of locality and power-counting in this context. Indeed, these usual space-time notions may not hold for neural networks (since inputs can be arbitrary), however, the renormalization group provides natural notions of locality and scaling. Moreover, we comment on several subtleties, for example, that data components may not have a permutation symmetry: in that case, we argue that random tensor field theories could provide a natural generalization. Second, we improve the perturbative Wilsonian renormalization from [1] by providing an analysis in terms of the nonperturbative renormalization group using the Wetterich-Morris equation. An important difference with usual nonperturbative RG analysis is that only the effective (IR) 2-point function is known, which requires setting the problem with care. Our aim is to provide a useful formalism to investigate neural networks behavior beyond the large-width limit (i.e. far from Gaussian limit) in a nonperturbative fashion. A major result of our analysis is that changing the standard deviation of the neural network weight distribution can be interpreted as a renormalization flow in the space of networks. We focus on translations invariant kernels and provide preliminary numerical results.

Disorder Averaging and its UV (Dis)Contents

“…This is related to the broader question of how well a low energy observer can ever hope to reconstruct a given UV completion, see, e.g.,[14,[40][41][42][43][44] as well as[45][46][47][48][49][50][51][52].…”

Machine Learning Symmetry

“…More generally, we also note that ideas and methods from mathematical physics, in particular quantum field theory, have been used to explicate the dynamics of neural networks (see e.g. [49,50,51,52]). Indeed, the very structures of quantum field theory and holography appear to be closely connected to certain neural networks [53].…”

The edge of chaos: quantum field theory and deep neural networks

“…Another RG-inspired approach appeared in the earlier work [7], which was further developed in two more papers [8,9] that appeared while this project was underway (see also [41]). As alluded above, [7] explores the relation between QFTs and the Gaussian process (i.e., N → ∞) limit of neural networks in detail.…”

“…In particular, they posit an initially Gaussian action, and show that the deviations from the infinite-width limit can be accurately modelled by the addition of a quartic interaction term, whose contribution is suppressed by 1/N . The irrelevance of higher (e.g., six-point) interactions can be understood from an RG perspective, which was developed more thoroughly in [8]. These authors also introduced the phrase "neural network phenomenology" to emphasize the fact that the form of the action in all of the above approaches is posited on general grounds, with couplings determined either by experiment or by tracking effective interactions.…”

“…Recently, physicists have begun exploring the parallels between deep neural networks and quantum field theory, leading to a number of interesting works on what is sometimes called the NN-QFT correspondence [2,[5][6][7][8][9]. 1 One key aspect of this approach is the fact that many modern architectures admit a Gaussian process limit: as the number of neurons (per layer, i.e., the width) N → ∞, the network is described by a Gaussian distribution, and hence may be modelled as a free (non-interacting) quantum field theory.…”