Variational Autoencoder with Learned Latent Structure

Abstract

The manifold hypothesis states that high-dimensional data can be modeled as lying on or near a low-dimensional, nonlinear manifold. Variational Autoencoders (VAEs) approximate this manifold by learning mappings from low-dimensional latent vectors to high-dimensional data while encouraging a global structure in the latent space through the use of a specified prior distribution. When this prior does not match the structure of the true data manifold, it can lead to a less accurate model of the data. To resolve this mismatch, we introduce the Variational Autoencoder with Learned Latent Structure (VAELLS) which incorporates a learnable manifold model into the latent space of a VAE. This enables us to learn the nonlinear manifold structure from the data and use that structure to define a prior in the latent space. The integration of a latent manifold model not only ensures that our prior is well-matched to the data, but also allows us to define generative transformation paths in the latent space and describe class manifolds by transformations stemming from examples of each class. We validate our model on examples with known latent structure and also demonstrate its capabilities on a real-world dataset. 1

Variational Sparse Coding with Learned Thresholding

“…The Laplacian prior with a fixed threshold performs the best among variational methods on objective (9). We note that methods that learn the threshold parameters tend to increase the number of active latent features, leading to an increase in L1 penalty in the validation loss.…”

“…Rather than give each sample equal probability, we introduce a new sampling strategy motivated by the approximation to expectations utilized in [37,9]. In these works, it is observed that the selected prior distribution concentrates most of its probability mass around the maximum value.…”

“…We start in this simplified setting since it allows us to compare against baseline sparse coding strategies (that use MAP estimates) and so we can analyze the dictionary we learn for each method. As a baseline, we infer coefficients z k using the FISTA algorithm [5] to optimize objective (9). This objective includes a data fidelity term, sparsity-encouraging 1 penalty on latent features, and a Frobenius norm regularizer on the dictionary to prevent unbounded growth:…”

“…Performance of different inference methods with a linear generator on whitened image patches over three trials. Validation loss computed on objective(9). Multi-information measures the statistical efficiency of inferred codes.…”