Tighter Variational Bounds are Not Necessarily Better

Rainforth, Tom; Kosiorek, Adam R.; Lê, Tuấn Anh; Maddison, Chris J.; Igl, Maximilian; Wood, Frank; Teh, Yee Whye

doi:10.48550/arxiv.1802.04537

Cited by 17 publications

(32 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During training we used a K=100 importance weighted auto-encoder (IWAE) estimator for gradient computation (Burda et al, 2015). We implemented a doubly reparameterised gradient estimator (DReG) for IWAE, which has recently been shown to yield lower-variance gradient estimates than the gradient computed naively by reversemode automatic differentiation (Tucker et al, 2018), while also avoiding the known problem of poor gradients for the variational approximation with greater numbers of importance samples (Rainforth et al, 2018). We found that the DReG estimator gave modest improvements for the black-box, but very significant improvements in performance and rate of convergence for the white-box (see Appendix figure 8).…”

Section: Architecture and Optimisation Detailsmentioning

confidence: 99%

Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems

Roeder¹,

Grant²,

Phillips³

et al. 2019

Preprint

View full text Add to dashboard Cite

We introduce a flexible, scalable Bayesian inference framework for nonlinear dynamical systems characterised by distinct and hierarchical variability at the individual, group, and population levels. Our model class is a generalisation of nonlinear mixed-effects (NLME) dynamical systems, the statistical workhorse for many experimental sciences. We cast parameter inference as stochastic optimisation of an end-to-end differentiable, block-conditional variational autoencoder. We specify the dynamics of the data-generating process as an ordinary differential equation (ODE) such that both the ODE and its solver are fully differentiable. This model class is highly flexible: the ODE right-hand sides can be a mixture of userprescribed or "white-box" sub-components and neural network or "black-box" sub-components. Using stochastic optimisation, our amortised inference algorithm could seamlessly scale up to massive data collection pipelines (common in labs with robotic automation). Finally, our framework supports interpretability with respect to the underlying dynamics, as well as predictive generalization to unseen combinations of group components (also called "zero-shot" learning). We empirically validate our method by predicting the dynamic behaviour of bacteria that were genetically engineered to function as biosensors.

show abstract

Section: Architecture and Optimisation Detailsmentioning

confidence: 99%

Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems

Roeder¹,

Grant²,

Phillips³

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…( 3). This is an important reason why tighter variational bounds are not necessarily better (Rainforth et al, 2018) and we therefore don't use it in place of the ELBO as the objective function. Some of other graphical models, such as the variational Gaussian mixture model (Attias, 1999), have the similar tighter lower bounds.…”

Section: Data Dependent Expectation (Dde)mentioning

confidence: 99%

Variational Inference for Sparse Gaussian Process Modulated Hawkes Process

Zhang,

Walder,

Rizoiu

2019

Preprint

View full text Add to dashboard Cite

The Hawkes process has been widely applied to modeling self-exciting events, including neuron spikes, earthquakes and tweets. To avoid designing parametric kernel functions and to be able to quantify the prediction confidence, non-parametric Bayesian Hawkes processes have been proposed. However the inference of such models suffers from unscalability or slow convergence. In this paper, we first propose a new non-parametric Bayesian Hawkes process whose triggering kernel is modeled as a squared sparse Gaussian process. Second, we present the variational inference scheme for the model optimization, which has the advantage of linear time complexity by leveraging the stationarity of the triggering kernel. Third, we contribute a tighter lower bound than the evidence lower bound of the marginal likelihood for the model selection. Finally, we exploit synthetic data and large-scale social media data to validate the efficiency of our method and the practical utility of our approximate marginal likelihood. We show that our approach outperforms state-of-the-art non-parametric Bayesian and non-Bayesian methods.

show abstract

“…Recent work [19] has shown that optimizing the importance weighted bound can degrade the overall learning process of the inference network because the signal-to-noise ratio of the gradient zi|x) is the gradient estimate of η). The SN R k (φ) converges to 0 for inference network as k → ∞, and the gradient estimates of φ become completely random.…”

Section: Iw-avb and Iw-aaementioning

confidence: 99%

Importance Weighted Adversarial Variational Autoencoders for Spike Inference from Calcium Imaging Data

Im,

Prakhya,

Yan

et al. 2019

Preprint

View full text Add to dashboard Cite

The Importance Weighted Auto Encoder (IWAE) objective has been shown to improve the training of generative models over the standard Variational Auto Encoder (VAE) objective. Here, we derive importance weighted extensions to AVB and AAE. These latent variable models use implicitly defined inference networks whose approximate posterior density q φ (z|x) cannot be directly evaluated, an essential ingredient for importance weighting. We show improved training and inference in latent variable models with our adversarially trained importance weighting method, and derive new theoretical connections between adversarial generative model training criteria and marginal likelihood based methods. We apply these methods to the important problem of inferring spiking neural activity from calcium imaging data, a challenging posterior inference problem in neuroscience, and show that posterior samples from the adversarial methods outperform factorized posteriors used in VAEs.

show abstract

Tighter Variational Bounds are Not Necessarily Better

Cited by 17 publications

References 0 publications

Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems

Efficient Amortised Bayesian Inference for Hierarchical and Nonlinear Dynamical Systems

Variational Inference for Sparse Gaussian Process Modulated Hawkes Process

Importance Weighted Adversarial Variational Autoencoders for Spike Inference from Calcium Imaging Data

Contact Info

Product

Resources

About