Toward a theory of machine learning

Abstract

We define a neural network as a septuple consisting of (1) a state vector, (2) an input projection, (3) an output projection, (4) a weight matrix, (5) a bias vector, (6) an activation map and (7) a loss function. We argue that the loss function can be imposed either on the boundary (i.e. input and/or output neurons) or in the bulk (i.e. hidden neurons) for both supervised and unsupervised systems. We apply the principle of maximum entropy to derive a canonical ensemble of the state vectors subject to a constraint imposed on the bulk loss function by a Lagrange multiplier (or an inverse temperature parameter). We show that in an equilibrium the canonical partition function must be a product of two factors: a function of the temperature, and a function of the bias vector and weight matrix. Consequently, the total Shannon entropy consists of two terms which represent, respectively, a thermodynamic entropy and a complexity of the neural network. We derive the first and second laws of learning: during learning the total entropy must decrease until the system reaches an equilibrium (i.e. the second law), and the increment in the loss function must be proportional to the increment in the thermodynamic entropy plus the increment in the complexity (i.e. the first law). We calculate the entropy destruction to show that the efficiency of learning is given by the Laplacian of the total free energy, which is to be maximized in an optimal neural architecture, and explain why the optimization condition is better satisfied in a deep network with a large number of hidden layers. The key properties of the model are verified numerically by training a supervised feedforward neural network using the stochastic gradient descent method. We also discuss a possibility that the entire Universe at its most fundamental level is a neural network.

RETRACTED: Multiple Sclerosis Recognition by Biorthogonal Wavelet Features and Fitness-Scaled Adaptive Genetic Algorithm

“…The N F features are sent to a feedforward neural network (FNN), a type of artificial neural network wherein the connections do not generate a loop (Vanchurin, 2021). Within FNN, the information moves along one direction-forward-from input layers through hidden layers to output layer (Valizadeh et al, 2021).…”

Towards a Theory of Quantum Gravity from Neural Networks

“…(See ref. [ 27 ] for details.) There are two types of degrees of freedom: non-trainable variables (or the bias vector and weight matrix ) and non-trainable variables (or the state of boundary and bulk neurons).…”

“…The main objective of learning is to find the trainable variables (or the bias vector and weight matrix ) which minimize the time-average of some loss function. For example, the boundary loss function is and the bulk loss function can be defined as where in addition to the first term, which represents a sum of local errors over all neurons, there may be a second term which represents either local objectives or constraints imposed by a neural architecture [ 27 ]. Note that the boundary loss is usually used in supervised learning, but the bulk loss may be used for both supervised and unsupervised learning tasks.…”

“…To develop a statistical description of learning [ 27 ], consider a joint probability distribution over both trainable and non-trainable variables, where and denote, respectively, the marginal and conditional distributions. If the non-trainable variables quickly equilibrate, then their distribution must be given by the maximum entropy distribution [ 28 , 29 ], where is a Lagrange multiplier which imposes a constraint on the loss function, …”

Survey on Technique and User Profiling in Unsupervised Machine Learning Method

“…In data mining technique, deep learning is used to actionable insights from unstructured information. Machine Learning (ML) [1] is "the study of computer techniques to systematize the process of information accumulation from instances." It is classified into two types: supervised and Unsupervised Machine Learning (UML).…”