Identifiable Deep Generative Models via Sparse Decoding

Moran, Gemma E.; Sridhar, Dhanya; Wang, Yixin; Blei, David M.

doi:10.48550/arxiv.2110.10804

Cited by 5 publications

(8 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These include Laplacian [2], Spike-and-Slab [52], and Beta-Bernoulli [50] distributions. Other work has incorporated sparsity through hierarchical posterior distributions [47] or through group-sparsity in connections to generator networks [1,34]. We refer the reader to Zhang et al [58] for a review of variational inference.…”

Section: Related Workmentioning

confidence: 99%

Variational Sparse Coding with Learned Thresholding

Fallah¹,

Rozell²

2022

Preprint

View full text Add to dashboard Cite

Sparse coding strategies have been lauded for their parsimonious representations of data that leverage low dimensional structure. However, inference of these codes typically relies on an optimization procedure with poor computational scaling in high-dimensional problems. For example, sparse inference in the representations learned in the high-dimensional intermediary layers of deep neural networks (DNNs) requires an iterative minimization to be performed at each training step. As such, recent, quick methods in variational inference have been proposed to infer sparse codes by learning a distribution over the codes with a DNN. In this work, we propose a new approach to variational sparse coding that allows us to learn sparse distributions by thresholding samples, avoiding the use of problematic relaxations. We first evaluate and analyze our method by training a linear generator, showing that it has superior performance, statistical efficiency, and gradient estimation compared to other sparse distributions. We then compare to a standard variational autoencoder using a DNN generator on the Fashion MNIST and CelebA datasets.Preprint. Under review.

show abstract

Section: Related Workmentioning

confidence: 99%

Variational Sparse Coding with Learned Thresholding

Fallah¹,

Rozell²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Our approach is to avoid additional forms of supervision altogether, and enforce identifiability in a purely unsupervised fashion. Recent work along these lines includes , who propose to use Brenier maps and input convex neural networks, and Moran et al (2021) who leverage sparsity and an anchor feature assumption. Aside from different assumptions, the main difference between this line of work and our work is that their work only identifies the latent space P (Z), whereas our focus is on jointly identifying both P (Z) and f .…”

Section: Related Workmentioning

confidence: 99%

“…To the best of our knowledge, our results are the first to establish identifiability of both the latent space and decoder for deep generative models without conditioning in the latent space or weak supervision. We note that and Moran et al (2021) also propose deep architectures that identify the latent space, but not the decoder.…”

Section: Generative Modelmentioning

confidence: 99%

Identifiability of deep generative models under mixture priors without auxiliary information

Kivva¹,

Rajendran²,

Ravikumar³

et al. 2022

Preprint

View full text Add to dashboard Cite

We prove identifiability of a broad class of deep latent variable models that (a) have universal approximation capabilities and (b) are the decoders of variational autoencoders that are commonly used in practice. Unlike existing work, our analysis does not require weak supervision, auxiliary information, or conditioning in the latent space. Recently, there has been a surge of works studying identifiability of such models. In these works, the main assumption is that along with the data, an auxiliary variable u (also known as side information) is observed as well. At the same time, several works have empirically observed that this doesn't seem to be necessary in practice. In this work, we explain this behavior by showing that for a broad class of generative (i.e. unsupervised) models with universal approximation capabilities, the side information u is not necessary: We prove identifiability of the entire generative model where we do not observe u and only observe the data x. The models we consider are tightly connected with autoencoder architectures used in practice that leverage mixture priors in the latent space and ReLU/leaky-ReLU activations in the encoder. Our main result is an identifiability hierarchy that significantly generalizes previous work and exposes how different assumptions lead to different "strengths" of identifiability. For example, our weakest result establishes (unsupervised) identifiability up to an affine transformation, which already improves existing work. It's well known that these models have universal approximation capabilities and moreover, they have been extensively used in practice to learn representations of data.

show abstract

“…Importantly, as many functions can perfectly fit the training data-i.e. the function represented by the DNN model is often nonidentifiable (Raue et al, 2013;Martín & González, 2010;Moran et al, 2021), and given an input x the uncertainty estimate incorporates quantification if possible hypothesis function "agree" on the prediction of x. It is thus natural to consider that training with perturbations that maximize this quantity will most efficiently narrow down the hypothesis set, leaving out the functions that increase the margin.…”

Section: Introductionmentioning

confidence: 99%

Improving Generalization via Uncertainty Driven Perturbations

Pagliardini¹,

Manunza²,

Jäggi³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently Shah et al., 2020 pointed out the pitfalls of the simplicity bias-the tendency of gradientbased algorithms to learn simple models-which include the model's high sensitivity to small input perturbations, as well as sub-optimal margins. In particular, while Stochastic Gradient Descent yields max-margin boundary on linear models, such guarantee does not extend to nonlinear models. To mitigate the simplicity bias, we consider uncertainty-driven perturbations (UDP) of the training data points, obtained iteratively by following the direction that maximizes the model's estimated uncertainty. The uncertainty estimate does not rely on the input's label and it is highest at the decision boundary, and-unlike loss-driven perturbations-it allows for using a larger range of values for the perturbation magnitude. Furthermore, as real-world datasets have non-isotropic distances between data points of different classes, the above property is particularly appealing for increasing the margin of the decision boundary, which in turn improves the model's generalization. We show that UDP is guaranteed to achieve the maximum margin decision boundary on linear models and that it notably increases it on challenging simulated datasets. For nonlinear models, we show empirically that UDP reduces the simplicity bias and learns more exhaustive features. Interestingly, it also achieves competitive loss-based robustness and generalization trade-off on several datasets. Our codebase is available here: github.com/mpagli/Uncertainty-Driven-Perturbations

show abstract

Identifiable Deep Generative Models via Sparse Decoding

Cited by 5 publications

References 14 publications

Variational Sparse Coding with Learned Thresholding

Variational Sparse Coding with Learned Thresholding

Identifiability of deep generative models under mixture priors without auxiliary information

Improving Generalization via Uncertainty Driven Perturbations

Contact Info

Product

Resources

About