Dirichlet Variational Autoencoder

Joo, Weonyoung; Lee, Wonsung; Park, Sungrae; Moon, Il‐Chul

doi:10.48550/arxiv.1901.02739

Cited by 4 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Building on the original VAE (Kingma & Welling, 2013), Nalisnick et al (2016) utilise a latent mixture of Gaussians, aiming to capture class structure in an unsupervised fashion, and propose a Bayesian non-parametric prior, further developed in (Nalisnick & Smyth, 2017). Similarly, Joo et al (2019) suggest a Dirichlet posterior in latent space to avoid some of the previously observed component-collapsing phenomena. Lastly, Jiang et al (2017) propose Variational Deep Embedding (VaDE) focused on the goal of clustering in an i.i.d setting.…”

Section: Related Workmentioning

confidence: 97%

“…Interestingly, the result is also better than DGR, suggesting that by holistically incorporating the generative process and classifier into the same model, and focusing on the broader unsupervised, task-agnostic perspective, CURL is still effective in the supervised domain. (Kingma & Welling, 2013), DirichletVAE (Joo et al, 2019), SBVAE (Nalisnick & Smyth, 2017), and VaDE (Jiang et al, 2017). We utilise the same architecture and hyperparameter settings as in Joo et al (2019) for consistency, with latent spaces of dimension 50 and 100 for MNIST and Omniglot respectively; and full details of the experimental setup can be found in Appendix C.3.…”

Section: External Benchmarksmentioning

confidence: 99%

See 1 more Smart Citation

Continual Unsupervised Representation Learning

Rao,

Visin,

Rusu

et al. 2019

Preprint

View full text Add to dashboard Cite

Continual learning aims to improve the ability of modern learning systems to deal with non-stationary distributions, typically by attempting to learn a series of tasks sequentially. Prior art in the field has largely considered supervised or reinforcement learning tasks, and often assumes full knowledge of task labels and boundaries. In this work, we propose an approach (CURL) to tackle a more general problem that we will refer to as unsupervised continual learning. The focus is on learning representations without any knowledge about task identity, and we explore scenarios when there are abrupt changes between tasks, smooth transitions from one task to another, or even when the data is shuffled. The proposed approach performs task inference directly within the model, is able to dynamically expand to capture new concepts over its lifetime, and incorporates additional rehearsal-based techniques to deal with catastrophic forgetting. We demonstrate the efficacy of CURL in an unsupervised learning setting with MNIST and Omniglot, where the lack of labels ensures no information is leaked about the task. Further, we demonstrate strong performance compared to prior art in an i.i.d setting, or when adapting the technique to supervised tasks such as incremental class learning.

show abstract

Section: Related Workmentioning

confidence: 97%

Section: External Benchmarksmentioning

confidence: 99%

Continual Unsupervised Representation Learning

Rao,

Visin,

Rusu

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The expectation values for the sampled vector components are r i = α i / j α j , so as a prior it will create a hierarchy among different mixture components. In our application, it imposes a compact latent space, whose latent dimensions can be interpreted as mixture weights in a multinomial mixture model [49,50].…”

Section: Dirichlet-vaementioning

confidence: 99%

“…We then study a generalisation, in which the Gaussian prior is replaced by a Gaussian-Mixture prior (GM-VAE) [42][43][44][45][46][47][48] in Section 3. In Section 4 we introduce the Dirichlet-VAE (DVAE), which uses a compact latent space with a Dirichlet prior [49,50]. Through a specific choice of decoder architecture we can interpret the decoder weights as the parameters of the mixture distributions in the probabilistic model, and can visualise these to directly interpret what the neural network is learning.…”

Section: Introductionmentioning

confidence: 99%

Better Latent Spaces for Better Autoencoders

Dillon,

Plehn,

Sauer

et al. 2021

Preprint

View full text Add to dashboard Cite

Autoencoders as tools behind anomaly searches at the LHC have the structural problem that they only work in one direction, extracting jets with higher complexity but not the other way around. To address this, we derive classifiers from the latent space of (variational) autoencoders, specifically in Gaussian mixture and Dirichlet latent spaces. In particular, the Dirichlet setup solves the problem and improves both the performance and the interpretability of the networks.

show abstract

“…For example, Stick-Breaking VAE (Nalisnick & Smyth, 2017) assumes a Griffiths-Engen-McCloskey (GEM) prior (Pitman, 2002), and the authors utilized the Kumaraswamy distribution (Kumaraswamy, 1980) to approximate the Beta distribution used by the GEM distribution. Dirichlet VAE (Joo et al, 2019) assumes a Dirichlet prior, and the authors utilized the approximation by the inverse Gamma cumulative density function (Knowles, 2015) and the composition of Gamma random variables to form the Dirichlet distribution.…”

Section: Introductionmentioning

confidence: 99%

Generalized Gumbel-Softmax Gradient Estimator for Generic Discrete Random Variables

Joo¹,

Kim²,

Shin³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Estimating the gradients of stochastic nodes is one of the crucial research questions in the deep generative modeling community. This estimation problem becomes further complex when we regard the stochastic nodes to be discrete because pathwise derivative techniques can not be applied. Hence, the gradient estimation requires the score function methods or the continuous relaxation of the discrete random variables. This paper proposes a general version of the Gumbel-Softmax estimator with continuous relaxation, and this estimator is able to relax the discreteness of probability distributions, including broader types than the current practice. In detail, we utilize the truncation of discrete random variables and the Gumbel-Softmax trick with a linear transformation for the relaxation. The proposed approach enables the relaxed discrete random variable to be reparameterized and to backpropagate through a large scale stochastic neural network. Our experiments consist of synthetic data analyses, which show the efficacy of our methods, and topic model analyses, which demonstrates the value of the proposed estimation in practices.

show abstract

Dirichlet Variational Autoencoder

Cited by 4 publications

References 0 publications

Continual Unsupervised Representation Learning

Continual Unsupervised Representation Learning

Better Latent Spaces for Better Autoencoders

Generalized Gumbel-Softmax Gradient Estimator for Generic Discrete Random Variables

Contact Info

Product

Resources

About