Nonlinear Dimension Reduction by PDF Estimation

Baggenstoss, Paul M.; Kay, S.

doi:10.1109/tsp.2022.3151317

Cited by 7 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It may be more enlightening to ask what does the AFs say about the input data. Any monotonically increasing function can be seen as an estimator of the input distribution [6]. This view that AFs are PDF estimators can be best described mathematically with the change of variables theorem.…”

Section: B Theoretical Justificationmentioning

confidence: 99%

“…The projected belief network (PBN) is a generative network that is based on PDF estimation, a direct extension of ( 2), (4) to dimension-reducing transformations [6], so it is the ideal paradigm to test the concepts of TCA. The PBN is based on the idea of back-projection through a given feed-forward neural network (FFNN), a way to reconstruct or re-sample the input data based on the network output [17].…”

Section: A Descriptionmentioning

confidence: 99%

See 1 more Smart Citation

Trainable Compound Activation Functions for Machine Learning

Baggenstoss¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Activation functions (AF) are necessary components of neural networks that allow approximation of functions, but AFs in current use are usually simple monotonically increasing functions. In this paper, we propose trainable compound AF (TCA) composed of a sum of shifted and scaled simple AFs. TCAs increase the effectiveness of networks with fewer parameters compared to added layers. TCAs have a special interpretation in generative networks because they effectively estimate the marginal distributions of each dimension of the data using a mixture distribution, reducing modality and making linear dimension reduction more effective. When used in restricted Boltzmann machines (RBMs), they result in a novel type of RBM with mixture-based stochastic units. Improved performance is demonstrated in experiments using RBMs, deep belief networks (DBN), projected belief networks (PBN), and variational autoencoders (VAE).

show abstract

Section: B Theoretical Justificationmentioning

confidence: 99%

Section: A Descriptionmentioning

confidence: 99%

Trainable Compound Activation Functions for Machine Learning

Baggenstoss¹

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…It has recently been shown that the information at the output of a dimension-reducing transformation can be maximized using probability density function projection (PDF projection) [1]. PDF projection estimates the distribution of the input data, simultaneously with a dimension-reducing transformation that extracts the latent variables [2], [3].…”

Section: A Motivation: Advantages Of Pbnmentioning

confidence: 99%

Using the Projected Belief Network at High Dimensions

Baggenstoss¹

2022

Preprint

View full text Add to dashboard Cite

The projected belief network (PBN) is a layered generative network (LGN) with tractable likelihood function, and is based on a feed-forward neural network (FFNN). There are two versions of the PBN: stochastic and deterministic (D-PBN), and each has theoretical advantages over other LGNs. However, implementation of the PBN requires an iterative algorithm that includes the inversion of a symmetric matrix of size M × M in each layer, where M is the layer output dimension. This, and the fact that the network must be always dimension-reducing in each layer, can limit the types of problems where the PBN can be applied. In this paper, we describe techniques to avoid or mitigate these restrictions and use the PBN effectively at high dimension. We apply the discriminatively aligned PBN (PBN-DA) to classifying and auto-encoding high-dimensional spectrograms of acoustic events. We also present the discriminatively aligned D-PBN for the first time.

show abstract