Parallelizing MCMC for Bayesian spatiotemporal geostatistical models

Yan, Jun; Cowles, Mary Kathryn; Wang, Shaowen; Armstrong, Marc P.

doi:10.1007/s11222-007-9022-2

Cited by 44 publications

(28 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Blackford et al, 1996) using OpenMP or MPI. Yan et al (2007) demonstrate that PLAPACK based block matrix algorithms provide similar factors of improvement to the OpenMP results described in this article. As the focus of this paper is primarily on software which runs on a single physical machine (with a few processors / cores), we have not run simulations which use MPI.…”

supporting

confidence: 64%

“…The slice sampler (Damien et al, 1999;Mira and Tierney, 2002;Neal, 1997Neal, , 2003a has been proposed as an easily implemented method for constructing an MCMC algorithm and can, in many circumstances, result in samplers with good mixing properties. Slice sampling has been used in many contexts, for example in spatial models (Agarwal and Gelfand, 2005;Yan et al, 2007), in biological models (Lewis et al, 2005;Shahbaba and Neal, 2006;Sun et al, 2007), variable selection (Kinney and Dunson, 2007;Nott and Leonte, 2004), and machine learning (Andrieu et al, 2003;Kovac, 2005;Mackay, 2002). Slice samplers can adapt to local characteristics of the distribution, which can make them easier to tune than Metropolis-Hastings approaches.…”

Section: Introductionmentioning

confidence: 99%

“…While parallel computing has been explored in a few other MCMC contexts, the focus appears to have been primarily on three areas: (1) parallelizing likelihood evaluations, for example, by distributing matrix computations (cf. Whiley and Wilson, 2004;Yan et al, 2007), (2) exploiting conditional independence to facilitate simultaneous updates of parameters (cf. Wilkinson, 2005), and (3) utilizing multiple Markov chains (Rosenthal, 2000).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Parallel multivariate slice sampling

2010

View full text Add to dashboard Cite

Slice sampling provides an easily implemented method for constructing a Markov chain Monte Carlo (MCMC) algorithm. However, slice sampling has two major drawbacks: (i) it requires repeated evaluation of likelihoods for each update, which can make it impractical when evaluations are expensive or as the number of evaluations grows (geometrically) with the dimension of the slice sampler, and (ii) since it can be challenging to construct multivariate updates, the updates are typically univariate, which often results in slow mixing samplers. We propose an approach to multivariate slice sampling that naturally lends itself to a parallel implementation. Our approach takes advantage of recent advances in computer architectures, for instance, the newest generation of graphics cards can execute roughly 30, 000 threads simultaneously. We demonstrate that it is possible to construct a multivariate slice sampler that has good mixing properties and is efficient in terms of computing time. The contributions of this article are therefore twofold. We study approaches for constructing a multivariate slice sampler, and we show how parallel computing can be useful for making MCMC algorithms computationally efficient. We study various implementations of our algorithm in the context of real and simulated data.

show abstract

supporting

confidence: 64%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Parallel multivariate slice sampling

2010

View full text Add to dashboard Cite

show abstract

“…More recent work combines the use of multiple chains with adaptive MCMC in an attempt to use these multiple sources of information to learn an appropriate proposal distribution (12,13). Sometimes, specific MCMC algorithms are directly amenable to parallelization, such as independent Metropolis−Hastings (14) or slice sampling (15), as indeed are some statistical models via careful reparameterization (16) or implementation on specialist hardware, such as graphics processing units (GPUs) (17,18); however, these approaches are often problem specific and not generally applicable. For problems involving large amounts of data, parallelization may in some cases also be possible by partitioning the data and analyzing each subset using standard MCMC methods simultaneously on multiple machines (19).…”

mentioning

confidence: 99%

A general construction for parallelizing Metropolis−Hastings algorithms

Calderhead

2014

Proc. Natl. Acad. Sci. U.S.A.

117

130

View full text Add to dashboard Cite

Markov chain Monte Carlo methods (MCMC) are essential tools for solving many modern-day statistical and computational problems; however, a major limitation is the inherently sequential nature of these algorithms. In this paper, we propose a natural generalization of the Metropolis−Hastings algorithm that allows for parallelizing a single chain using existing MCMC methods. We do so by proposing multiple points in parallel, then constructing and sampling from a finite-state Markov chain on the proposed points such that the overall procedure has the correct target density as its stationary distribution. Our approach is generally applicable and straightforward to implement. We demonstrate how this construction may be used to greatly increase the computational speed and statistical efficiency of a variety of existing MCMC methods, including Metropolis-Adjusted Langevin Algorithms and Adaptive MCMC. Furthermore, we show how it allows for a principled way of using every integration step within Hamiltonian Monte Carlo methods; our approach increases robustness to the choice of algorithmic parameters and results in increased accuracy of Monte Carlo estimates with little extra computational cost. S ince its introduction in the 1970s, the Metropolis−Hastings algorithm has revolutionized computational statistics (1). The ability to draw samples from an arbitrary probability distribution, πðXÞ, known only up to a constant, by constructing a Markov chain that converges to the correct stationary distribution has enabled the practical application of Bayesian inference for modeling a huge variety of scientific phenomena, and has resulted in Metropolis−Hastings being noted as one of the top 10 most important algorithms from the 20th century (2). Despite regular increases in available computing power, Markov chain Monte Carlo (MCMC) algorithms can still be computationally very expensive; many thousands of iterations may be necessary to obtain low-variance estimates of the required quantities with an oftentimes complex statistical model being evaluated for each set of proposed parameters. Furthermore, many Metropolis−Hastings algorithms are severely limited by their inherently sequential nature.Many approaches have been proposed for improving the statistical efficiency of MCMC, and although such algorithms are guaranteed to converge asymptotically to the stationary distribution, their performance over a finite number of iterations can vary hugely. Much research effort has therefore focused on developing transition kernels that enable moves to be proposed far from the current point and subsequently accepted with high probability, taking into account, for example, the correlation structure of the parameter space (3, 4), or by using Hamiltonian dynamics (5) or diffusion processes (6). A recent investigation into proposal kernels suggests that more exotic distributions, such as the Bactrian kernel, might also be used to increase the statistical efficiency of MCMC algorithms (7). The efficient exploration of high-dimensional and multimo...

show abstract

“…They are widely used in statistical applications, ranging from machine learning [1], [2], [3], statistical physics [4], [5] and geostatistics [6] to medical imaging [7], genetics [8], phylogenetics [9], computational biology [10], [11] and stochastic optimization [12], [13]. Sampling from a distribution is fundamental in these applications because we are typically interested in performing the following task:…”

Section: Introductionmentioning

confidence: 99%

Population-Based MCMC on Multi-Core CPUs, GPUs and FPGAs

Mingas

Bouganis

2016

IEEE Trans. Comput.

View full text Add to dashboard Cite

Abstract-Markov Chain Monte Carlo (MCMC) is a method to draw samples from a given probability distribution. Its frequent use for solving probabilistic inference problems, where big-scale data are repeatedly processed, means that MCMC runtimes can be unacceptably large. This paper focuses on population-based MCMC, a popular family of computationally intensive MCMC samplers; we propose novel, highly optimized accelerators in three parallel hardware platforms (multi-core CPUs, GPUs and FPGAs), in order to address the performance limitations of sequential software implementations. For each platform, we jointly exploit the nature of the underlying hardware and the special characteristics of population-based MCMC. We focus particularly on the use of custom arithmetic precision, introducing two novel methods which employ custom precision in the largest part of the algorithm in order to reduce runtime, without causing sampling errors. We apply these methods to all platforms. The FPGA accelerators are up to 114x faster than multi-core CPUs and up to 53x faster than GPUs when doing inference on mixture models.

show abstract

Parallelizing MCMC for Bayesian spatiotemporal geostatistical models

Cited by 44 publications

References 25 publications

Parallel multivariate slice sampling

Parallel multivariate slice sampling

A general construction for parallelizing Metropolis−Hastings algorithms

Population-Based MCMC on Multi-Core CPUs, GPUs and FPGAs

Contact Info

Product

Resources

About