Towards Deeper Understanding of Variational Autoencoding Models

Zhao, Shengjia; Song, Jiaming; Ermon, Stefano

doi:10.48550/arxiv.1702.08658

Cited by 73 publications

(75 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Despite these advantages, followup works have identified a few important drawbacks of VAEs. The VAE objective is at the risk of posterior collapse -learning a latent space distribution which is independent of the input distribution if the KL term dominates the reconstruction term Zhao et al, 2017;. The poor sample qualities of VAE has been attributed to a mismatch between the prior (which is used for drawing samples) and the posterior Tomczak & Welling, 2018;Dai & Wipf, 2019;Bauer & Mnih, 2019).…”

Section: Related Workmentioning

confidence: 99%

“…However, they come with some challenges. For instance, VAEs suffer from the posterior collapse problem Zhao et al, 2017;, and a mismatch between the posterior and prior distribution Tomczak & Welling, 2018;Dai & Wipf, 2019;Bauer & Mnih, 2019). GANs are known to have the mode collapse problem (Che et al, 2016;Dumoulin et al, 2016;Donahue et al, 2016) and optimization instability (Arjovsky & Bottou, 2017) due to their saddle point problem formulation.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE

Arpit¹,

Bhatnagar²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Wasserstein autoencoder (WAE) shows that matching two distributions is equivalent to minimizing a simple autoencoder (AE) loss under the constraint that the latent space of this AE matches a pre-specified prior distribution. This latent space distribution matching is a core component of WAE, and a challenging task. In this paper, we propose to use the contrastive learning framework that has been shown to be effective for self-supervised representation learning, as a means to resolve this problem. We do so by exploiting the fact that contrastive learning objectives optimize the latent space distribution to be uniform over the unit hyper-sphere, which can be easily sampled from. We show that using the contrastive learning framework to optimize the WAE loss achieves faster convergence and more stable optimization compared with existing popular algorithms for WAE. This is also reflected in the FID scores on CelebA and CIFAR-10 datasets, and the realistic generated image quality on the CelebA-HQ dataset.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE

Arpit¹,

Bhatnagar²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…a fixed Gaussian distribution), therefore the model distribution is absolutely continuous and maximum likelihood learning is thus well defined. However, common distributions such as natural images usually don't have Gaussian observational noise [46]. Therefore, we focus on modeling distributions that lie on a low-dimensional manifold.…”

Section: Related Workmentioning

confidence: 99%

Spread Flows for Manifold Modelling

Zhang¹,

Sun²,

Zhang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Flow-based generative models typically define a latent space with dimensionality identical to the observational space. In many problems, however, the data does not populate the full ambient data-space that they natively reside in, rather inhabiting a lower-dimensional manifold. In such scenarios, flow-based models are unable to represent data structures exactly as their density will always have support off the data manifold, potentially resulting in degradation of model performance. In addition, the requirement for equal latent and data space dimensionality can unnecessarily increase complexity for contemporary flow models. Towards addressing these problems, we propose to learn a manifold prior that affords benefits to both sample generation and representation quality. An auxiliary benefit of our approach is the ability to identify the intrinsic dimension of the data distribution. * The work was done during an internship in Huawei Noah's Ark Lab. 2 The distribution is absolutely continuous with respect to the Lebesgue measure. In this case, it has a density function, see [12].Preprint. Under review.

show abstract

“…Images reconstructed from encoder-decoder models are usually blurry. Blurry images are due to using the mean squared error (MSE) loss between the reconstructed and original image, implying the error between both images is Gaussian noise, which is not in the case in seismic images (Zhao et al, 2017).…”

Section: Adversarial Trainingmentioning

confidence: 99%

Self-Supervised Delineation of Geological Structures using Orthogonal Latent Space Projection

Aribido¹,

AlRegib²,

Alaudah³

2021

Preprint

View full text Add to dashboard Cite

We developed two machine learning frameworks that could assist in automated litho-stratigraphic interpretation of seismic volumes without any manual hand labeling from an experienced seismic interpreter. The first framework is an unsupervised hierarchical clustering model to divide seismic images from a volume into certain number of clusters determined by the algorithm. The clustering framework uses a combination of density and hierarchical techniques to determine the size and homogeneity of the clusters. The second framework consists of a self-supervised deep learning framework to label regions of geological interest in seismic images. It projects the latent-space of an encoderdecoder architecture unto two orthogonal subspaces, from which it learns to delineate regions of interest in the seismic images. To demonstrate an application of both frameworks, a seismic volume was clustered into various contiguous clusters, from which four clusters were selected based on distinct seismic patterns: horizons, faults, salt domes and chaotic structures. Images from the selected clusters are used to train the encoderdecoder network. The output of the encoder-decoder network is a probability map of the possibility an amplitude reflection event belongs to an interesting geological structure. The structures are delineated using the probability map. The delineated images are further used to post-train a segmentation model to extend our results to full-vertical sections. The results on vertical sections show that we can factorize a seismic volume into its corresponding structural components. Lastly, we showed that our deep learning framework could be modeled as an attribute extractor and we compared our attribute result with various existing attributes in literature and demonstrate competitive performance with them.

show abstract

Towards Deeper Understanding of Variational Autoencoding Models

Cited by 73 publications

References 2 publications

Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE

Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE

Spread Flows for Manifold Modelling

Self-Supervised Delineation of Geological Structures using Orthogonal Latent Space Projection

Contact Info

Product

Resources

About