Preventing Posterior Collapse with Levenshtein Variational Autoencoder

Havrylov, Serhii; Titov, Ivan

doi:10.48550/arxiv.2004.14758

Cited by 3 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We propose using a random schedule for the variable β in order to reduce the potential effects that could be caused by the posterior collapse problem in VAEs (Lucas et al, 2019 ; Havrylov and Titov, 2020 ; Takida et al, 2021 ) and to maintain a balance with the reconstruction loss. We take a sample from a uniform distribution

for every example that we go through in the training process.…”

Section: Methodsmentioning

confidence: 99%

Feature alignment as a generative process

Farias

Maziero²

2023

Front. Artif. Intell.

View full text Add to dashboard Cite

Reversibility in artificial neural networks allows us to retrieve the input given an output. We present feature alignment, a method for approximating reversibility in arbitrary neural networks. We train a network by minimizing the distance between the output of a data point and the random output with respect to a random input. We applied the technique to the MNIST, CIFAR-10, CelebA, and STL-10 image datasets. We demonstrate that this method can roughly recover images from just their latent representation without the need of a decoder. By utilizing the formulation of variational autoencoders, we demonstrate that it is possible to produce new images that are statistically comparable to the training data. Furthermore, we demonstrate that the quality of the images can be improved by coupling a generator and a discriminator together. In addition, we show how this method, with a few minor modifications, can be used to train networks locally, which has the potential to save computational memory resources.

show abstract

for every example that we go through in the training process.…”

Section: Methodsmentioning

confidence: 99%

Feature alignment as a generative process

Farias

Maziero²

2023

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…The posterior collapse effect is a known problem of shallow VAEs when some or even all of the latent variables do not carry any information about the observed data. There are various methods to deal with this issue for VAEs, such as changing the parameterization [Dieng et al, 2019, He et al, 2019, changing the optimization or the objective [Alemi et al, 2018, Bowman et al, 2016, Fu et al, 2019, Havrylov and Titov, 2020, Razavi et al, 2019, or using hierarchical models [Child, 2021, Maaløe et al, 2017, Tomczak and Welling, 2018, Vahdat and Kautz, 2020b. Here, we focus entirely on the hierarchical VAEs since the posterior collapse problem is not fully analyzed in their context.…”

Section: An Analysis Of the Posterior Collapse In Hierarchical Vaesmentioning

confidence: 99%

Analyzing the Posterior Collapse in Hierarchical Variational Autoencoders

Kuzina¹,

Tomczak²

2023

Preprint

View full text Add to dashboard Cite

Hierarchical Variational Autoencoders (VAEs) are among the most popular likelihood-based generative models. There is rather a consensus that the top-down hierarchical VAEs allow to effectively learn deep latent structures and avoid problems like the posterior collapse. Here, we show that it is not necessarily the case and the problem of collapsing posteriors remains. To discourage the posterior collapse, we propose a new deep hierarchical VAE with a partly fixed encoder, specifically, we use Discrete Cosine Transform to obtain top latent variables. In a series of experiments, we observe that the proposed modification allows us to achieve better utilization of the latent space. Further, we demonstrate that the proposed approach can be useful for compression and robustness to adversarial attacks.

show abstract

“…Kingma et al [13] and Pelsmaeker and Aziz [19] proposed to introduce constraints to force the divergence term to be greater than a hyperparameter. Finally, alternative surrogate objectives to train VAEs in the context of text generation have been proposed [16,9].…”

Section: Introductionmentioning

confidence: 99%