Out-of-equilibrium dynamical mean-field equations for the perceptron model

Abstract

Perceptrons are the building blocks of many theoretical approaches to a wide range of complex systems, ranging from neural networks and deep learning machines, to constraint satisfaction problems, glasses and ecosystems. Despite their applicability and importance, a detailed study of their Langevin dynamics has never been performed yet. Here we derive the mean-field dynamical equations that describe the continuous random perceptron in the thermodynamic limit, in a very general setting with arbitrary noise and friction kernels, not necessarily related by equilibrium relations. We derive the equations in two ways: via a dynamical cavity method, and via a path-integral approach in its supersymmetric formulation. The end point of both approaches is the reduction of the dynamics of the system to an effective stochastic process for a representative dynamical variable. Because the perceptron is formally very close to a system of interacting particles in a high dimensional space, the methods we develop here can be transferred to the study of liquid and glasses in high dimensions.Potentially interesting applications are thus the study of the glass transition in active matter, the study of the dynamics around the jamming transition, and the calculation of rheological properties in driven systems. Contents

Out-of-equilibrium dynamical equations of infinite-dimensional particle systems I. The isotropic case

“…[65], and in the same spirit as what has been done in Ref. [75] for the continuous random perceptron. It relies on the following two main ideas:…”

“…Of course, the saddle-point equation (89) is the same as the one obtained with our Gaussian approximation in Eq. (45), as it can be self-consistently checked by using the different rescalings (62)-(63) and (75). Consequently, it also provides a closure relation for the variance α(a, b), the rescaled counterpart of Eq.…”

Gradient descent dynamics and the jamming transition in infinite dimensions

“…In this section, we recapitulate the DMFT equations for infinite-dimensional particle systems, as derived in [24][25][26][27][28]. We focus on the specific case of GD dynamics of soft repulsive spheres, see e.g.…”

“…The aim of this work is to investigate to some extent the dynamical mean field theory (DMFT) equations [3,5,6,9,10,[22][23][24][25][26][27][28] that describe GD dynamics in mean-field complex systems, which display a jamming transition. We will focus in particular on the infinite-dimensional limit of soft repulsive particles [27], and we will thus use the jamming terminology in the main text.…”

“…In the appendix, in order to show the generality of the approach, we discuss the single-layer fully-connected perceptron [26,31,33], which is one of the simplest data classifiers in supervised machine learning. We show how the results of the main text can be translated to that case.…”