Mean-field variational approximate Bayesian inference for latent variable models

Consonni, Guido; Marin, Jean‐Michel

doi:10.1016/j.csda.2006.10.028

Cited by 47 publications

(33 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further trials (not presented here) on the data of Section V, on synthetic truth data and on more complex/highly sparse benchmark data (e.g., the Forensic Glass set [3]) suggest that the proposed mean field VB approximations can perform badly with very sparse data sets and are instead most reliable for intermediate and large sample sizes (i.e., relative to the number of parameters to be estimated). This agrees with the findings of previous empirical and theoretical studies on the asymptotic properties of mean field VB approximations in other related models [15], [18], [35], which show that VB estimates are generally biased with respect to the true posterior statistics (as an unavoidable consequence of 'deliberate model misspecification' via the mean field assumption). While such biases can be quite significant at small sample sizes, they generally become much less significant as the proportion of observed to unobserved variables in the training set increases (or equivalently, as the number of training data increases when the number of hidden parameters can be fixed, as in the models studied here).…”

Section: A Performance Considerationssupporting

confidence: 91%

See 1 more Smart Citation

Variational Bayesian Learning of Probabilistic Discriminative Models With Latent Softmax Variables

Ahmed

Campbell

2011

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

This paper presents new variational Bayes (VB) approximations for learning probabilistic discriminative models with latent softmax variables, such as subclass-based multimodal softmax and mixture of experts models. The VB approximations derived here lead to closed-form approximate parameter posteriors and suitable metrics for model selection. Unlike other Bayesian methods for this challenging class of models, the proposed VB methods require neither restrictive structural assumptions nor sampling approximations to cope with the problematic softmax function. As such, the proposed VB methods are also easily extendable to more complex softmax-based hierarchical discriminative models and regression models (for continuous outputs). The proposed VB methods are evaluated on benchmark classification data and a decision modeling application, demonstrating good results.

show abstract

Section: A Performance Considerationssupporting

confidence: 91%

“…From (5), the posterior over the hidden variables takes exactly the same form as (15), where the denominator is now given by marginalization of (34) over the variables. Once again, substitution of (35)-(40) into (15) leads to an intractable posterior, as in the MMS case.…”

Section: Variational Bayes Learning For Me Modelsmentioning

confidence: 98%

Variational Bayesian Learning of Probabilistic Discriminative Models With Latent Softmax Variables

Ahmed

Campbell

2011

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

show abstract

“…This is clearly seen in the centre graphs where the θ i values where i > p have been "switched off". The results in both examples are similar in that the variational posterior noise distribution is more compact than the actual, a tendency reported by a number of authors (for example [23], [24], [25]), the estimated θ values are similar to the actuals, with a tendency to be underestimated, and the reconstructions of the data are good.…”

Section: A Synthesising Datasupporting

confidence: 71%

Robust Autoregression: Student-t Innovations Using Variational Bayes

Christmas

Everson

2011

IEEE Trans. Signal Process.

View full text Add to dashboard Cite

Abstract-Autoregression (AR) is a tool commonly used to understand and predict time series data. Traditionally the excitation noise is modelled as a Gaussian. However, real-world data may not be Gaussian in nature, and it is known that Gaussian models are adversely affected by the presence of outliers. We introduce a Bayesian AR model in which the excitation noise is assumed to be Student-t distributed. Variational Bayesian approximations to the posterior distributions of the model parameters are used to overcome the intractable integrations inherent in the Bayesian model. Independent Automatic Relevance Determination (ARD) priors over each of the AR coefficients are used to estimate the model order.Using synthetic data we show that the Student-t model performs well against both Gaussian and leptokurtic data, in terms of parameter estimation (including the model order), and is much more robust to outliers than either Gaussian or finite mixtures of Gaussians models.We apply the model to strongly leptokurtic EEG signals and show that the Student-t model makes more accurate one-stepahead predictions than the Gaussian model and provides more consistent estimates of the AR coefficients over simultaneously recorded EEG channels.

show abstract

“…Various algorithms for VAEs (Kingma & Welling, ; Maddison et al, ; Jang et al, ) can be used to optimize this objective. In these VAE algorithms, mean‐field family (Blei & Jordan, ; Consonni & Marin, ) is assumed for the working model Q ( Z | X , θ ), that is, Q ( Z | X , θ ), can be factorized with respect to Z . Moreover, Q ( X | Z , f ) can also be factorized across components of X , which is consistent with our generating scheme that assumes X | Z is mutually independent.…”

Section: A Deep Knockoff Generator Through Latent Variablesmentioning

confidence: 99%

Deep latent variable models for generating knockoffs

Ying

Zheng

2019

Stat

View full text Add to dashboard Cite

Selective inference is an emerging field in big data analytics; it targets on conducting variable selection and providing statistical inference at the same time. Among various selective inference frameworks, the model‐X framework offers the most flexible tool to equip almost any machine learning method with the ability for false discovery rate (FDR) controlled variable selection. This paper provides a practical and flexible approach to generate knockoffs. We propose to fit a latent variable model for generating knockoffs. Under general conditions, the knockoffs can be generated by approximate inference of a latent variable, which captures all the correlation of predictors. We propose an algorithm based on recent advancement in stochastic variational inference to approximately reconstruct the distribution of data via the latent variables. We demonstrate that our proposed method can achieve FDR control and better power than existing knockoff generation methods in various simulated settings and a real data example for finding mutations associated with drug resistance in human immunodeficiency virus type 1 patients.

show abstract

Mean-field variational approximate Bayesian inference for latent variable models

Cited by 47 publications

References 9 publications

Variational Bayesian Learning of Probabilistic Discriminative Models With Latent Softmax Variables

Variational Bayesian Learning of Probabilistic Discriminative Models With Latent Softmax Variables

Robust Autoregression: Student-t Innovations Using Variational Bayes

Deep latent variable models for generating knockoffs

Contact Info

Product

Resources

About