GSNs : Generative Stochastic Networks

Alain, Guillaume; Bengio, Yoshua; Li, Yao; Yosinski, Jason; Thibodeau-Laufer, Éric; Zhang, Saizheng; Vincent, P.

doi:10.48550/arxiv.1503.05571

Cited by 5 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The use of deep neural networks as generative models for complex data has made great advances in recent years. This success has been achieved through a surprising diversity of training losses and model architectures, including denoising autoencoders (Vincent et al, 2010), variational autoencoders (Kingma & Welling, 2013;Rezende et al, 2014;Gregor et al, 2015;Kulkarni et al, 2015;Burda et al, 2015;Kingma et al, 2016), generative stochastic networks (Alain et al, 2015), diffusion probabilistic models (Sohl-Dickstein et al, 2015), autoregressive models (Theis & Bethge, 2015;van den Oord et al, 2016a;, real non-volume preserving transformations (Dinh et al, 2014;, Helmholtz machines (Dayan et al, 1995;Bornschein et al, 2015), and Generative Adversarial Networks (GANs) (Goodfellow et al, 2014).…”

Section: Introductionmentioning

confidence: 99%

Unrolled Generative Adversarial Networks

Metz¹,

Poole²,

Pfau³

et al. 2016

Preprint

189

329

View full text Add to dashboard Cite

We introduce a method to stabilize Generative Adversarial Networks (GANs) by defining the generator objective with respect to an unrolled optimization of the discriminator. This allows training to be adjusted between using the optimal discriminator in the generator's objective, which is ideal but infeasible in practice, and using the current value of the discriminator, which is often unstable and leads to poor solutions. We show how this technique solves the common problem of mode collapse, stabilizes training of GANs with complex recurrent generators, and increases diversity and coverage of the data distribution by the generator. * Work done as a member of the Google Brain Residency program (g.co/brainresidency) † Work completed as part of a Google Brain internship

show abstract

Section: Introductionmentioning

confidence: 99%

Unrolled Generative Adversarial Networks

Metz¹,

Poole²,

Pfau³

et al. 2016

Preprint

189

329

View full text Add to dashboard Cite

show abstract

“…Walkback and variational walkback Contrastive adjustment is closely related to the walkback algorithm in Generative Stochastic Networks (GSNs) [1]. GSNs form a Markov chain by alternating sampling from a corruption process and a denoising distribution.…”

Section: Related Workmentioning

confidence: 99%

Learning Stationary Markov Processes with Contrastive Adjustment

Bergenstråhle¹,

Lagergren²,

Lundeberg³

2023

Preprint

View full text Add to dashboard Cite

We introduce a new optimization algorithm, termed contrastive adjustment, for learning Markov transition kernels whose stationary distribution matches the data distribution. Contrastive adjustment is not restricted to a particular family of transition distributions and can be used to model data in both continuous and discrete state spaces. Inspired by recent work on noise-annealed sampling, we propose a particular transition operator, the noise kernel, that can trade mixing speed for sample fidelity. We show that contrastive adjustment is highly valuable in humancomputer design processes, as the stationarity of the learned Markov chain enables local exploration of the data manifold and makes it possible to iteratively refine outputs by human feedback. We compare the performance of noise kernels trained with contrastive adjustment to current state-of-the-art generative models and demonstrate promising results on a variety of image synthesis tasks.

show abstract

“…As a result, maximum reconstruction estimation (Hinton, Osindero, and Teh 2006;Vincent et al 2008;Bengio 2009;Vincent et al 2010;Socher et al 2011;Ammar, Dyer, and Smith 2014;Alain et al 2015) aims to find a set of parameters that maximize the product of reconstruction probabilities of the training data:…”

Section: Maximum Reconstruction Estimationmentioning

confidence: 99%

“…In this work, we introduce maximum reconstruction estimation (MRE) (Hinton, Osindero, and Teh 2006;Vincent et al 2008;Bengio 2009;Vincent et al 2010;Socher et al 2011;Ammar, Dyer, and Smith 2014;Alain et al 2015) for learning generative latent-variable models. The basic idea is to circumvent irrelevant but common correlations by maximizing the probability of reconstructing observed data.…”

Section: Introductionmentioning

confidence: 99%

Maximum Reconstruction Estimation for Generative Latent-Variable Models

Cheng

Liu

2017

AAAI

View full text Add to dashboard Cite

Generative latent-variable models are important for natural language processing due to their capability of providing compact representations of data. As conventional maximum likelihood estimation (MLE) is prone to focus on explaining irrelevant but common correlations in data, we apply maximum reconstruction estimation (MRE) to learning generative latent-variable models alternatively, which aims to find model parameters that maximize the probability of reconstructing the observed data. We develop tractable algorithms to directly learn hidden Markov models and IBM translation models using the MRE criterion, without the need to introduce a separate reconstruction model to facilitate efficient inference. Experiments on unsupervised part-of-speech induction and unsupervised word alignment show that our approach enables generative latent-variable models to better discover intended correlations in data and outperforms maximum likelihood estimators significantly.

show abstract

GSNs : Generative Stochastic Networks

Cited by 5 publications

References 0 publications

Unrolled Generative Adversarial Networks

Unrolled Generative Adversarial Networks

Learning Stationary Markov Processes with Contrastive Adjustment

Maximum Reconstruction Estimation for Generative Latent-Variable Models

Contact Info

Product

Resources

About