Maximum Entropy PDF Design Using Feature Density Constraints: Applications in Signal Processing

2018

Entropy

Self Cite

The maximum entropy principle introduced by Jaynes proposes that a data distribution should maximize the entropy subject to constraints imposed by the available knowledge. Jaynes provided a solution for the case when constraints were imposed on the expected value of a set of scalar functions of the data. These expected values are typically moments of the distribution. This paper describes how the method of maximum entropy PDF projection can be used to generalize the maximum entropy principle to constraints on the joint distribution of this set of functions.

“…It is in fact

It can be verified that ( 7 ) integrates to 1 on the manifold

. The entropy of ( 6 ) can be decomposed as the entropy of

plus the expected value of the entropy of the

(see Equation ( 8 ) in [ 8 ]):

Maximizing this quantity seems daunting, but there is one condition under which

has the maximum entropy for all

, and that is when

is the uniform distribution for all

. This, in turn, is achieved when

has a constant value on any manifold

.…”

Section: Resultsmentioning

confidence: 99%

Beyond Moments: Extending the Maximum Entropy Principle to Feature Distribution Constraints

2018

Entropy

Self Cite

“…This problem has been studied in detail and solutions exist for a wide range of features [8], [9], [10], [11], [5]. For the TED-TED RBM which uses the uniform reference distribution,…”

Section: Discussion and Implemetationmentioning

confidence: 99%

“…An ES is an optional scalar statistic, but insures the maximum entropy property of the PDF projection [11], [5]. As explained in [5], when the input data is limited to…”

Section: E Energy Statistic (Es)mentioning

confidence: 99%

See 1 more Smart Citation

Evaluating the RBM without integration using PDF projection

2017 25th European Signal Processing Conference (EUSIPCO)

2017

Self Cite

Abstract-In this paper, we apply probability density function (PDF) projection to arrive at an exact closed-form expression for the marginal distribution of the visible data of a restricted Boltzmann machine (RBM) without requiring integrating over the distribution of the hidden variables or needing to know the partition function. We express the visible data marginal as a projected PDF based on a set of sufficient statistics. When a Gaussian mixture model (GMM) is used to estimate the PDF of the sufficient statistics, then we arrive at a combined RBM/GMM model that serves as a general-purpose PDF estimator and Bayesian classifier. The approach extends recusively to compute the input distribution of a multi-layer network. We demonstrate the method using a reduced subset of the MNIST handwritten character data set.

“…To draw samples from (5), samples are drawn from the manifold M(z) with probability proportional to the value of the prior distribution p 0 (x). It can be shown [11] that the denominator in (5) can be written…”

Section: Manifold Distributionmentioning

confidence: 99%

A Neural Network Based on First Principles

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

2020

In this paper, a Neural network is derived from first principles, assuming only that each layer begins with a linear dimensionreducing transformation. The approach appeals to the principle of Maximum Entropy (MaxEnt) to find the posterior distribution of the input data of each layer, conditioned on the layer output variables. This posterior has a well-defined mean, the conditional mean estimator, that is calculated using a type of neural network with theoretically-derived activation functions similar to sigmoid, softplus, and relu. This implicitly provides a theoretical justification for their use. A theorem that finds the conditional distribution and conditional mean estimator under the MaxEnt prior is proposed, unifying results for special cases. Combining layers results in an autoencoder with conventional feed-forward analysis network and a type of linear Bayesian belief network in the reconstruction path.