Mode jumping MCMC for Bayesian variable selection in GLMM

Abstract

Generalized linear mixed models (GLMM) are used for inference and prediction in a wide range of different applications providing a powerful scientific tool. An increasing number of sources of data are becoming available, introducing a variety of candidate explanatory variables for these models. Selection of an optimal combination of variables is thus becoming crucial. In a Bayesian setting, the posterior distribution of the models, based on the observed data, can be viewed as a relevant measure for the model evidence. The number of possible models increases exponentially in the number of candidate variables. Moreover, the space of models has numerous local extrema in terms of posterior model probabilities. To resolve these issues a novel MCMC algorithm for the search through the model space via efficient mode jumping for GLMMs is introduced. The algorithm is based on 1 The corresponding author, Aliaksandr Hubin, is a PhD candidate at the University of Oslo, 0851 Moltke Moes vei 35 Oslo, Norway. Email: aliaksah@math.uio.no, tel: +4745171361. The paper has supplementary materials consisting of: R package: R package EMJMCMC to perform the efficient mode jumping MCMC described in the paper. (EMJMCMC 1.2.tar.gz; GNU zipped tar file). Data and code: Data (simulated and real) and R code for MJMCMC algorithm, postprocessing and creating figures wrapped together into a reference based EMJMCMC class.(code-and-data.zip; zip file containing the data, code and a read-me file (readme.pdf)) Details and Pseudo code: Proofs of the ergodicity of MJMCMC procedure, pseudo codes for MJMCMC and local combinatorial optimizers, parallelization strategies and some supplementary tables for the experiments. (appendix.pdf) that marginal likelihoods can be efficiently calculated within each model. It is recommended that either exact expressions or precise approximations of marginal likelihoods are applied. The suggested algorithm is applied to simulated data, the famous U.S. crime data, protein activity data and epigenetic data and is compared to several existing approaches.

Identifying groups of determinants in Bayesian model averaging using Dirichlet process clustering

“…In case of different groups of covariates being relevant, it is necessary for the sampler to visit the different modes corresponding to each of these groups in order for the proposed approach to be able to correctly identify the different groups in the postprocessing step. Alternative sampling approaches for BMA with a better mixing behavior, such as those proposed in Clyde et al (2011) and Hubin and Storvik (2018), might thus be preferable to be used in case of groups of relevant covariates being suspected. The impact of their use should be investigated in future work.…”

“…The posterior model weights for the visited models may then only be determined based on the marginal likelihood and the prior probabilities of the specified model prior, while the empirical frequencies of the visited models are not suitable measures. Further alternative sampling schemes with better mixing have also been proposed, see, for example, Clyde et al (2011) and Hubin and Storvik (2018). In the following empirical analysis, only the algorithm is employed.…”

A Novel Algorithmic Approach to Bayesian Logic Regression (with Discussion)

“…The success of GMJMCMC relies upon resolving the local extrema issue, which is mainly achieved by combining the following two ideas. First, when iterating through a fixed search space S, GMJMCMC utilizes the MJMCMC algorithm (Hubin and Storvik, 2016a) which was specifically constructed to explore multi-modal regression spaces efficiently. Second, the evolution of the search spaces is governed within the framework of a genetic algorithm where a population consists of a finite number of trees forming the current search space.…”

“…Clearly the genetic algorithm used to update search spaces results in a Markov chain of model spaces. In the future it will be interesting to generalize the mode jumping ideas from Hubin and Storvik (2016a) to the Markov chain of search spaces, making it converge to the right limiting distribution in the joint space of models, parameters and search spaces, whilst remaining the property of not getting stuck in local modes.…”

“…The main contribution of this paper is the proposed search algorithm, named GMJMCMC, which provides a better search strategy for exploring the model space than previous approaches. GMJMCMC combines genetic algorithm ideas with the mode jumping Markov Chain Monte Carlo (MJMCMC) algorithm (Hubin and Storvik, 2016a) in order to be able to jump between local modes in the model space. After formally introducing logic regression and describing the GMJM-CMC algorithm in detail we will present results from a comprehensive simulation study.…”