Bayesian graph convolutional neural networks via tempered MCMC

Abstract

Deep learning models, such as convolutional neural networks, have long been applied to image and multi-media tasks, particularly those with structured data. More recently, there has been more attention to unstructured data that can be represented via graphs. These types of data are often found in health and medicine, social networks, and research data repositories. Graph convolutional neural networks have recently gained attention in the field of deep learning that takes advantage of graph-based data representation with automatic feature extraction via convolutions. Given popularity of these methods in a wide range of applications, robust uncertainty quantification is vital. This remains a challenge for large models and unstructured datasets. Bayesian inference provides a principled and robust approach to uncertainty quantification of model parameters for deep learning models. Although Bayesian inference has been used extensively elsewhere, its application to deep learning remains limited due to the computational requirements of the Markov Chain Monte Carlo (MCMC) methods. Recent advances in parallel computing and advanced proposal schemes in sampling, such as incorporating gradients has allowed Bayesian deep learning methods to be implemented. In this paper, we present Bayesian graph deep learning techniques that employ state-of-art methods such as tempered MCMC sampling and advanced proposal schemes. Our results show that Bayesian graph convolutional methods can provide accuracy similar to advanced learning methods while providing a better alternative for robust uncertainty quantification for key benchmark problems.

Revisiting Bayesian Autoencoders With MCMC

“…Langevin-gradient proposal distribution that incorporate Gaussian noise with gradients for a single iteration (epoch). Recently, Langevin-gradient proposal distribution in MCMC sampling has been utilised for novel Bayesian neural learning methods [32,75,76,77].…”

A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications

“…Teye et al [71] showed that training deep models with batch normalization is equal to that of estimating the inference in Bayesian networks. Chandra et al [72] proposed Bayesian graph deep learning techniques that use MCMC samples with Langevin-gradient. Mandt et al [73] used SGD with a constant learning rate (constant SGD) to simulate the Markov chain with a stationary distribution and showed that constant SGD can approximate the posterior inference.…”

Hierarchical Graph Neural Network Based on Semi-Implicit Variational Inference

“…To overcome this drawback, Pal et.al [17] introduced a non-parametric BGCN, which uses the node features, training labels, and observed graph for posterior inference. Following this idea, many Bayesian-based approaches are proposed [18][19][20].…”