A Neural Network Based on First Principles

Abstract

In this paper, a Neural network is derived from first principles, assuming only that each layer begins with a linear dimensionreducing transformation. The approach appeals to the principle of Maximum Entropy (MaxEnt) to find the posterior distribution of the input data of each layer, conditioned on the layer output variables. This posterior has a well-defined mean, the conditional mean estimator, that is calculated using a type of neural network with theoretically-derived activation functions similar to sigmoid, softplus, and relu. This implicitly provides a theoretical justification for their use. A theorem that finds the conditional distribution and conditional mean estimator under the MaxEnt prior is proposed, unifying results for special cases. Combining layers results in an autoencoder with conventional feed-forward analysis network and a type of linear Bayesian belief network in the reconstruction path.

Using the Projected Belief Network at High Dimensions

“…The method is more general than other methods of non-linear dimension reduction (NLDR) [4], [5], [6], [7], [8]. Implementing PDF projection in a neural network architecture is called projected belief network (PBN) [9], [10], [11], [12]. As a result, the PBN is the most direct and general way to apply NLDR in a neural network architecture.…”

“…The deterministic PBN (D-PBN) can be used as an autoencoder [11], [9] and has theoretical advantage over conventional auto-encoders. While other auto-encoders use an empirical reconstruction network, the deterministic PBN reconstructs input data by backing up (back-projecting) through the same feed-forward neural network (FFNN) that was used to extract the features.…”

Maximum Entropy Auto-Encoding

“…Recently, it was shown how to solve for the conditional mean for data ranges and maximum entropy priors suitable for machine learning applications. The result is an auto-encoder called deterministic projected belief network (D-PBN) [8], [9], [10], [11].…”

“…The solution h always exists when z = W ′ x for some x ∈ X and is also known as the saddle point [9]. The MaxEnt activation, defined in (3), depends on X and the corresponding MaxEnt reference distribution, which takes the form (1) for a specific choice of a, b.…”

“…The MaxEnt reference hypotheses and the corresponding activation functions λ( ) are listed in Table I. Note that the TED and TG activation functions are similar in behavior to the sigmoid and softplus activations in common use, respectively (Figure 2 in [9]).…”