From neural PCA to deep unsupervised learning

Valpola, Harri

doi:10.48550/arxiv.1411.7783

Cited by 10 publications

(19 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Considering the bottom-up inference procedure and topdown generation procedure together, additional memory mechanisms can help to reduce the competition between invariant feature extraction and local variant reconstruction, especially when label information is provided (e.g., in supervised or semi-supervised setting). Similar idea is highlighted in the Ladder Network (Valpola, 2014;Rasmus et al, 2015), which reconstructs the input hierarchically using an extension of denoising autoencoders (dAEs) (Vincent et al, 2010) with the help of lateral connections and achieves excellent performance on semi-supervised learning (Rasmus et al, 2015). Though it is possible to interpret the Ladder Network probabilistically as in (Bengio et al, 2013b), we model the data likelihood directly with the help of external memory instead of explicit lateral edges.…”

Section: Related Workmentioning

confidence: 90%

“…We optimize the objective with a stochastic gradient variational Bayes (SGVB) method . Note that we cannot send the message of a intermediate layer in the recognition model to a layer in the generative model through a lateral connection as in Ladder Network (Valpola, 2014;Rasmus et al, 2015) because that indeed changes the distribution of p(x|z) according to the data x. However, we do not use any information of x in the generative model explicitly and the correctness of the variational bound can be verified.…”

Section: Q-net P-netmentioning

confidence: 99%

“…However, in practise, we found that a more flexible composition function can lead to a better result. Inspired by the Ladder Network (Valpola, 2014;Rasmus et al, 2015), we specify the combination function of h (i) m and h (i) g as element wise multiple layer perceptron with optionally final nonlinearity φ:…”

Section: Concrete Examples With Hierarchical Memory Mechanismsmentioning

confidence: 99%

See 2 more Smart Citations

Learning to Generate with Memory

Li,

Zhu,

Zhang

2016

Preprint

View full text Add to dashboard Cite

Memory units have been widely used to enrich the capabilities of deep networks on capturing long-term dependencies in reasoning and prediction tasks, but little investigation exists on deep generative models (DGMs) which are good at inferring high-level invariant representations from unlabeled data. This paper presents a deep generative model with a possibly large external memory and an attention mechanism to capture the local detail information that is often lost in the bottom-up abstraction process in representation learning. By adopting a smooth attention model, the whole network is trained end-to-end by optimizing a variational bound of data likelihood via auto-encoding variational Bayesian methods, where an asymmetric recognition network is learnt jointly to infer high-level invariant representations. The asymmetric architecture can reduce the competition between bottom-up invariant feature extraction and top-down generation of instance details. Our experiments on several datasets demonstrate that memory can significantly boost the performance of DGMs on various tasks, including density estimation, image generation, and missing value imputation, and DGMs with memory can achieve state-ofthe-art quantitative results.

show abstract

Section: Related Workmentioning

confidence: 90%

Section: Q-net P-netmentioning

confidence: 99%

Section: Concrete Examples With Hierarchical Memory Mechanismsmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Generate with Memory

Li,

Zhu,

Zhang

2016

Preprint

View full text Add to dashboard Cite

show abstract

“…Our approach is based on the recently proposed Ladder Network training procedure that has proven to be very successful, especially in cases where the training dataset is composed of both labeled and unlabeled data [9] [10]. In addition to the supervised objective, the Ladder Network also has an unsupervised objective corresponding to the reconstruction costs of a stack of denoising autoencoders.…”

Section: Introductionmentioning

confidence: 99%

A Semisupervised Approach for Language Identification based on Ladder Networks

Ben-Reuven¹,

Goldberger²

2016

The Speaker and Language Recognition Workshop (Odyssey 2016)

View full text Add to dashboard Cite

In this study we address the problem of training a neuralnetwork for language identification using both labeled and unlabeled speech samples in the form of i-vectors. We propose a neural network architecture that can also handle out-of-set languages. We utilize a modified version of the recently proposed Ladder Network semisupervised training procedure that optimizes the reconstruction costs of a stack of denoising autoencoders. We show that this approach can be successfully applied to the case where the training dataset is composed of both labeled and unlabeled acoustic data. The results show enhanced language identification on the NIST 2015 language identification dataset.

show abstract

“…3.2]. We note that the adopted architecture is similar to the Ladder network [23], where the clean pathway is used for prediction while the corrupted one guaranties that the network is noise-invariant. As done in [4], and in order to avoid over-fitting, we add a reconstruction loss function to our objectives MI-ADM in Eq.…”

mentioning

confidence: 99%

Deep Clustering: On the Link Between Discriminative Models and K-Means

Jabi

Pedersoli²,

Mitiche

et al. 2021

IEEE Trans. Pattern Anal. Mach. Intell.

102

View full text Add to dashboard Cite

In the context of recent deep clustering studies, discriminative models dominate the literature and report the most competitive performances. These models learn a deep discriminative neural network classifier in which the labels are latent. Typically, they use multinomial logistic regression posteriors and parameter regularization, as is very common in supervised learning. It is generally acknowledged that discriminative objective functions (e.g., those based on the mutual information or the KL divergence) are more flexible than generative approaches (e.g., K-means) in the sense that they make fewer assumptions about the data distributions and, typically, yield much better unsupervised deep learning results. On the surface, several recent discriminative models may seem unrelated to K-means. This study shows that these models are, in fact, equivalent to K-means under mild conditions and common posterior models and parameter regularization. We prove that, for the commonly used logistic regression posteriors, maximizing the L 2 regularized mutual information via an approximate alternating direction method (ADM) is equivalent to a soft and regularized K-means loss. Our theoretical analysis not only connects directly several recent state-of-the-art discriminative models to K-means, but also leads to a new soft and regularized deep K-means algorithm, which yields competitive performance on several image clustering benchmarks.

show abstract

From neural PCA to deep unsupervised learning

Cited by 10 publications

References 0 publications

Learning to Generate with Memory

Learning to Generate with Memory

A Semisupervised Approach for Language Identification based on Ladder Networks

Deep Clustering: On the Link Between Discriminative Models and K-Means

Contact Info

Product

Resources

About