Revisiting Bayesian Autoencoders With MCMC

Chandra, Rohitash; Jain, Mahir; Maharana, Manavendra; Krivitsky, Pavel N.

doi:10.1109/access.2022.3163270

Cited by 14 publications

(6 citation statements)

References 81 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, future research could test other ML models to capture the network structure in SMD. One potential avenue in this context -depending on the dataset dimensions -is the application of other probabilistic, bayesian latent space models, neural networks and autoencoders (e.g., Chandra et al, 2022;Radev et al, 2020;Yong & Brintrup, 2022). Such models provide the opportunity to explore the posterior distributions of opinion estimates, potentially leading to even more accurate predictions of sociopolitical outcomes.…”

Section: Discussionmentioning

confidence: 99%

To follow or not to follow - Estimating political opinion from Twitter data using network analysis and machine learning

Brandenstein,

Montag,

Sindermann

2024

Preprint

View full text Add to dashboard Cite

Studying public opinion stands as a fundamental pursuit for both policymakers and researchers. While traditional surveys remain the primary method to investigate individual political opinions, the advent of social media data (SMD) offers novel prospects. However, the number of studies using SMD to extract individuals’ political opinions are limited and differ greatly in their methodological approaches and levels of success. Recent studies highlight the benefits of analyzing individuals’ network structure to estimate political opinions. Nevertheless, current methodologies exhibit limitations, including the use of simplistic linear models that disregard the complexity of relationships within the network, as well as a predominant focus on the United States. Addressing these issues, we employ a Variational Autoencoder (VAE) machine learning model to extract individual opinion estimates from SMD of N = 276 008 German Twitter (now: X) users, compare its performance to a state-of-the-art linear model and validate model estimates on self-reported opinion measures. Our findings suggest that the VAE captures the network structure of Twitter users more accurately, leading to higher accuracy of predicting following decisions as well as correlations with self-reported political ideology and voting intentions. Our study emphasizes the necessity of advanced analytical approaches, capable of capturing complex relationships in social media networks when studying public opinion, at least in non-US contexts. This research expands the understanding of utilizing SMD for public opinion analysis and underscores the potency of machine learning techniques in enhancing the predictive accuracy of SMD.

show abstract

Section: Discussionmentioning

confidence: 99%

To follow or not to follow - Estimating political opinion from Twitter data using network analysis and machine learning

Brandenstein,

Montag,

Sindermann

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…In the case of Bayesian neural networks, the prior distribution can be based on the distribution of the weights and biases from similar neural network models. This can be seen as an example of expert knowledge and implemented in previous studies [69], [104]. Another example of expert knowledge is the concept of weight decay [105] regularisation (L2 or Ridge regression [106]) which restricts large weights and can be incorporated when defining the prior distribution (priors) [8], [8], [9].…”

Section: Mcmcmentioning

confidence: 99%

“…The method has shown to be effective for linear models [61] which motivated its use in Bayesian neural networks. In the literature, Langevin MCMC has been very promising for simple and deep neural networks [62], [69], [70]. Hence, we draw the proposed values for the parameters (θ p ) according to a one-step (epoch) gradient as shown in Equation 39.…”

Section: A: Langevin Proposal Distributionmentioning

confidence: 99%

“…These have the ability to provide a competitive alternative to stochastic gradient-descent [68] and Adam optimizers [6] with the addition of uncertainty quantification in predictions. These methods have also been applied to Bayesian deep learning models such as Bayesian autoencoders [69] and Bayesian graph convolutional neural networks (CNNs) [70] which require millions of trainable parameters to be represent as posterior distributions. Recently, Kapoor et al [66] combined tempered MCMC with particle swarm optimisationbased proposal distribution in a parallelized environment that showed more effective sampling when compared with the conventional approach.…”

Section: A Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bayesian Neural Networks via MCMC: A Python-Based Tutorial

Chandra,

Simmons

2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

Bayesian inference provides a methodology for parameter estimation and uncertainty quantification in machine learning and deep learning methods. Variational inference and Markov Chain Monte-Carlo (MCMC) sampling methods are used to implement Bayesian inference. In the past three decades, MCMC sampling methods have faced some challenges in being adapted to larger models (such as in deep learning) and big data problems. Advanced proposal distributions that incorporate gradients, such as a Langevin proposal distribution, provide a means to address some of the limitations of MCMC sampling for Bayesian neural networks. Furthermore, MCMC methods have typically been constrained to statisticians and currently not well-known among deep learning researchers. We present a tutorial for MCMC methods that covers simple Bayesian linear and logistic models, and Bayesian neural networks. The aim of this tutorial is to bridge the gap between theory and implementation via coding, given a general sparsity of libraries and tutorials to this end. This tutorial provides code in Python with data and instructions that enable their use and extension. We provide results for some benchmark problems showing the strengths and weaknesses of implementing the respective Bayesian models via MCMC. We highlight the challenges in sampling multi-modal posterior distributions for the case of Bayesian neural networks and the need for further improvement of convergence diagnosis methods.INDEX TERMS MCMC; Bayesian deep learning; Bayesian neural networks; Bayesian linear regression; Bayesian inference

show abstract

“…To resolve the computational burden arising from the hierarchical Bayesian model, we consolidate the Bayesian sampling procedure via Stochastic Gradient Langevin Dynamics (SGLD) with an adaptive empirical Bayesian variable selection method using Expectation-maximization. Instead of computing the full batch gradient, SGLD evaluates mini-batch gradients with injected random Gaussian noise, which is theoretically valid to generate Langevin-based proposal distribution [18,19]. The mini-batch gradient learning naturally fits into the training of deep neural networks and relaxes the scalability issue at the same time.…”

Section: Introductionmentioning

confidence: 99%

Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants

Mars Gao,

Nathan Kutz

2024

Proc. R. Soc. A.

View full text Add to dashboard Cite

Recent progress in autoencoder-based sparse identification of nonlinear dynamics (SINDy) under ℓ 1 constraints allows joint discoveries of governing equations and latent coordinate systems from spatio-temporal data, including simulated video frames. However, it is challenging for ℓ 1 -based sparse inference to perform correct identification for real data due to the noisy measurements and often limited sample sizes. To address the data-driven discovery of physics in the low-data and high-noise regimes, we propose Bayesian SINDy autoencoders, which incorporate a hierarchical Bayesian Spike-and-slab Gaussian Lasso prior. Bayesian SINDy autoencoder enables the joint discovery of governing equations and coordinate systems with uncertainty estimate. To resolve the challenging computational tractability of the Bayesian hierarchical setting, we adapt an adaptive empirical Bayesian method with Stochastic Gradient Langevin Dynamics (SGLD) which gives a computationally tractable way of Bayesian posterior sampling within our framework. Bayesian SINDy autoencoder achieves better physics discovery with lower data and fewer training epochs, along with valid uncertainty quantification suggested by the experimental studies. The Bayesian SINDy autoencoder can be applied to real video data, withaccurate physics discovery which correctly identifies the governing equation and provides a close estimate for standard physics constants like gravity g , for example, in videos of a pendulum.

show abstract

Revisiting Bayesian Autoencoders With MCMC

Cited by 14 publications

References 81 publications

To follow or not to follow - Estimating political opinion from Twitter data using network analysis and machine learning

To follow or not to follow - Estimating political opinion from Twitter data using network analysis and machine learning

Bayesian Neural Networks via MCMC: A Python-Based Tutorial

Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants

Contact Info

Product

Resources

About