Thermodynamic Neural Network

Abstract

This work describes a thermodynamically motivated neural network model that self-organizes to transport charge associated with internal and external potentials while in contact with a thermal bath. Isolated networks show multiscale dynamics and evidence of phase transitions, and externally driven networks evolve to efficiently connect external positive and negative potentials. The model implements techniques for rapid, global, reversible, conservative equilibration of node states followed by slow, local, irreversible, dissipative adaptation of the edge states as a means to create multiscale order. All interactions in the network are local and the network structures can be generic and recurrent. The model integrates concepts of fluctuation, dissipation, adaptation and equilibration and offers an illustration of the thermodynamic evolution of organization in open systems. The key conclusion of the work is that the dissipation of conserved physical quantities can drive the self-organization of open thermodynamic systems.

Molecular Learning of a Soft-Disks Fluid

“…It is no coincidence that these algorithms share strong similarities with Machine Learning. The formal analogy between thermodynamics and learning has a long story (Gori, Maggini et al 2016, Goldt and Seifert 2017, Hylton 2020. The literature describes several models that link thermodynamics to the conservation of information (the first law) and the relative entropy decrease (second law).…”

“…The literature describes several models that link thermodynamics to the conservation of information (the first law) and the relative entropy decrease (second law). These models range over from disordered lattice Ising-spin-glasses to Markovian processes and Neural Networks (Hylton 2020). In these last, kinetic and potential energy concepts interpret learning as a dissipation-driven adaptation mechanism.…”

“…In these last, kinetic and potential energy concepts interpret learning as a dissipation-driven adaptation mechanism. In other words, it is a selfemergent regularization process that works on a training set's approximation accuracy (Gori, Maggini et al 2016, Hylton 2020. Therefore, in Neural Networks, dynamics leading learning can be described as a dissipative process, in which the kinetic energy reflects the weight variations.…”

“…In contrast, the potential energy represents the penalty relating to the degree of satisfaction of the environmental constraints. (Gori, Maggini et al 2016, Hylton 2020.…”

“…For this reason, they are provided by sophisticated functions, like gradient descent, which simultaneously updates θ to find ' ( ), by calculating the partial derivative. Therefore, Neural Networks show conservation, dissipation, adaptation, and self-organization (Hylton 2020). Remarkably, all these properties belong to any other network as an open thermodynamic system (Hylton 2020).…”