Truncated Variational Expectation Maximization

Lücke, Jörg

doi:10.48550/arxiv.1610.03113

Cited by 9 publications

(33 citation statements)

References 34 publications

(112 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…which approximate the full posterior by truncating sums over the whole latent space to sums over subsets K n) which accumulate most of the posterior mass [51,38]. The subsets K (n) can then be update, e.g., using evolutionary algorithms with fitness defined to be a monotonic function of the model joint p( s, y | Θ).…”

Section: Appendix a Additional Details On Parameter Update Equationsmentioning

confidence: 99%

Maximal Causes for Exponential Family Observables

Mousavi,

Drefs,

Hirschberger

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

The data model of standard sparse coding assumes a weighted linear summation of latents to determine the mean of Gaussian observation noise. However, such a linear summation of latents is often at odds with non-Gaussian observables (e.g., means of the Bernoulli distribution have to lie in the unit interval), and also in the Gaussian case it can be difficult to justify for many types of data. Alternative superposition models (i.e., links between latents and observables) have therefore been investigated repeatedly. Here we show that using the maximum instead of a linear sum to link latents to observables allows for the derivation of very general and concise parameter update equations. Concretely, we derive a set of update equations that has the same functional form for all distributions of the exponential family (given that derivatives w.r.t. their parameters can be taken). Our results consequently allow for the development of latent variable models for commonly as well as for unusually distributed data. We numerically verify our analytical result assuming standard Gaussian, Gamma, Poisson, Bernoulli and Exponential distributions and point to some potential applications.

show abstract

Section: Appendix a Additional Details On Parameter Update Equationsmentioning

confidence: 99%

Maximal Causes for Exponential Family Observables

Mousavi,

Drefs,

Hirschberger

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Centrally for this work, truncated posteriors allow a specific alternative reformulation of the bound that enables efficient optimization. The reformulation recombines the entropy term of the original form (2) with the first expectation value into a single term, and is given by (see Lücke, 2019, for details):…”

Section: Truncated Variational Optimizationmentioning

confidence: 99%

“…For our choice of variational distributions, it is not trivial that the entropy term actually can be ignored because the encoding model q Φ ( z; x) in (3) is defined in terms of the decoding model and its parameters. For truncated distributions, however, it can be shown that the entropy term can still be ignored(Lücke, 2019).…”

mentioning

confidence: 99%

Direct Evolutionary Optimization of Variational Autoencoders With Binary Latents

Guiraud¹,

Drefs²,

Lücke³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Discrete latent variables are considered important to model the generation process of real world data, which has motivated research on Variational Autoencoders (VAEs) with discrete latents. However, standard VAE training is not possible in this case, which has motivated different strategies to manipulate discrete distributions in order to train discrete VAEs similarly to conventional ones. Here we ask if it is also possible to keep the discrete nature of the latents fully intact by applying a direct discrete optimization for the encoding model. The studied approach is consequently strongly diverting from standard VAE training by altogether sidestepping absolute standard VAE mechanisms such as sampling approximation, reparameterization trick and amortization. Discrete optimization is realized in a variational setting using truncated posteriors in conjunction with evolutionary algorithms (using a recently suggested approach). For VAEs with binary latents, we first show how such a discrete variational method (A) ties into gradient ascent for network weights and (B) uses the decoder network to select latent states for training. More conventional amortized training is, as may be expected, more efficient than direct discrete optimization, and applicable to large neural networks. However, we here find direct optimization to be efficiently scalable to hundreds of latent variables using smaller networks. More importantly, we find the effectiveness of direct optimization to be highly competitive in 'zero-shot' learning (where high effectiveness for small networks is required). In contrast to large supervised neural networks, the here investigated VAEs can, e.g., denoise a single image without previous training on clean data and/or training on large image datasets. More generally, the studied approach shows that training of VAEs is indeed possible without sampling-based approximation and reparameterization, which may be interesting for the analysis of VAE-training in general. In the regime of few data, direct optimization, furthermore, makes VAEs competitive for denoising where they have previously been outperformed by non-generative approaches.

show abstract

“…Sheikh et al, 2014Sheikh et al, , 2019, we here apply a fully variational approach, i.e., we define a variational loop to optimize the variational parameters. Based on the specific functional form of truncated posteriors, optimal variational parameters Λ = (K, Θ) are given by setting Θ = Θ and by seeking K which optimize (see Lücke, 2019, for details):…”

Section: Evolutionary Optimization Of Truncated Posteriorsmentioning

confidence: 99%

Evolutionary Variational Optimization of Generative Models

Drefs¹,

Guiraud²,

Lücke³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

We combine two popular optimization approaches to derive learning algorithms for generative models: variational optimization and evolutionary algorithms. The combination is realized for generative models with discrete latents by using truncated posteriors as the family of variational distributions. The variational parameters of truncated posteriors are sets of latent states. By interpreting these states as genomes of individuals and by using the variational lower bound to define a fitness, we can apply evolutionary algorithms to realize the variational loop. The used variational distributions are very flexible and we show that evolutionary algorithms can effectively and efficiently optimize the variational bound. Furthermore, the variational loop is generally applicable ("black box") with no analytical derivations required. To show general applicability, we apply the approach to three generative models (we use noisy-OR Bayes Nets, Binary Sparse Coding, and Spike-and-Slab Sparse Coding). To demonstrate effectiveness and efficiency of the novel variational approach, we use the standard competitive benchmarks of image denoising and inpainting. The benchmarks allow quantitative comparisons to a wide range of methods including probabilistic approaches, deep deterministic and generative networks, and non-local image processing methods. In the category of "zero-shot" learning (when only the corrupted image is used for training), we observed the evolutionary variational algorithm to significantly improve the state-of-the-art in many benchmark settings. For one well-known inpainting benchmark, we also observed state-of-the-art performance across all categories of algorithms although we only train on the corrupted image. In general, our investigations highlight the importance of research on optimization methods for generative models to achieve performance improvements.

show abstract

Truncated Variational Expectation Maximization

Cited by 9 publications

References 34 publications

Maximal Causes for Exponential Family Observables

Maximal Causes for Exponential Family Observables

Direct Evolutionary Optimization of Variational Autoencoders With Binary Latents

Evolutionary Variational Optimization of Generative Models

Contact Info

Product

Resources

About