On the Effect of the Form of the Posterior Approximation in Variational Learning of ICA Models

Ilin, Alexander; Valpola, Harri

doi:10.1007/s11063-005-5265-0

Cited by 15 publications

(17 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main reason for using this suboptimal process is practical: even with a non-Gaussian source prior the factorial source posterior approximation would not allow determining the proper rotation of the independent sources [34]. This could be corrected by more advanced approximation techniques [5].…”

Section: Discussionmentioning

confidence: 99%

“…The disadvantage of HNFA is that the approximation of the posterior density is farther away from the true posterior density. This may occasionally lead to inferior performance [34]. The linear shortcut is included in the model to partially help this, because representing nearly linear mappings would otherwise be significantly more difficult than with the MLP model.…”

Section: S(t) H(t) X(t)mentioning

confidence: 99%

“…The NIFA model with its fully factorial posterior approximation seems to suffer from similar problems of not being able to extract the true independent sources as linear algorithms based on similar principles [34].…”

Section: On Different Source Modelsmentioning

confidence: 99%

See 2 more Smart Citations

Blind separation of nonlinear mixtures by variational Bayesian learning

Honkela¹,

Valpola²,

Ilin³

et al. 2007

Digital Signal Processing

Self Cite

View full text Add to dashboard Cite

Blind separation of sources from nonlinear mixtures is a challenging and often illposed problem. We present three methods for solving this problem: an improved nonlinear factor analysis (NFA) method using multilayer perceptron (MLP) network to model the nonlinearity, a hierarchical NFA (HNFA) method suitable for larger problems and a post-nonlinear NFA (PNFA) method for more restricted post-nonlinear mixtures. The methods are based on variational Bayesian learning, which provides the needed regularisation and allows for easy handling of missing data. While the basic methods are incapable of recovering the correct rotation of the source space, they can discover the underlying nonlinear manifold and allow reconstruction of the original sources using standard linear independent component analysis (ICA) techniques.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: S(t) H(t) X(t)mentioning

confidence: 99%

See 1 more Smart Citation

Blind separation of nonlinear mixtures by variational Bayesian learning

Honkela¹,

Valpola²,

Ilin³

et al. 2007

Digital Signal Processing

Self Cite

View full text Add to dashboard Cite

show abstract

“…To elaborate more, regardless of whatever the priors are, stochastic variables have independent representational aptitude and thus can add information at the time of reconstruction. In stark contrast, deterministic variables cannot add any information at reconstruction time [16].…”

Section: B Stochastic Latent Variable Models and Deterministic Autoementioning

confidence: 99%

Ladder Networks: Learning under Massive Label Deficit

Mirza¹,

Syed²,

Memon³

et al. 2017

ijacsa

View full text Add to dashboard Cite

Abstract-Advancement in deep unsupervised learning are finally bringing machine learning close to natural learning, which happens with as few as one labeled instance. Ladder Networks are the newest deep learning architecture that proposes semi-supervised learning at scale. This work discusses how the ladder network model successfully combines supervised and unsupervised learning taking it beyond the pre-training realm. The model learns from the structure, rather than the labels alone transforming it from a label learner to a structural observer. We extend the previously-reported results by lowering the number of labels, and report an error of 1.27 on 40 labels only, on the MNIST dataset that in a fully supervised setting, uses 60000 labeled training instances.

show abstract

“…For instance Ilin and Valpola (2003) have shown that it can compromise the quality of separation in ICA. This finding is relevant since we are using similar types of latent variable models with linear mappings.…”

Section: Consequenses Of the Posterior Approximationmentioning

confidence: 99%

Hierarchical models of variance sources

2004

Self Cite

View full text Add to dashboard Cite

In many models, variances are assumed to be constant although this assumption is often unrealistic in practice. Joint modelling of means and variances is difficult in many learning approaches, because it can lead into infinite probability densities. We show that a Bayesian variational technique which is sensitive to probability mass instead of density is able to jointly model both variances and means. We consider a model structure where a Gaussian variable, called variance node, controls the variance of another Gaussian variable. Variance nodes make it possible to build hierarchical models for both variances and means. We report experiments with artificial data which demonstrate the ability of the learning algorithm to find variance sources explaining and characterizing well the variances in the multidimensional data. Experiments with biomedical MEG data show that variance sources are present in real-world signals.

show abstract

On the Effect of the Form of the Posterior Approximation in Variational Learning of ICA Models

Cited by 15 publications

References 13 publications

Blind separation of nonlinear mixtures by variational Bayesian learning

Blind separation of nonlinear mixtures by variational Bayesian learning

Ladder Networks: Learning under Massive Label Deficit

Hierarchical models of variance sources

Contact Info

Product

Resources

About