Scalable inference for crossed random effects models

Abstract

Summary We develop methodology and complexity theory for Markov chain Monte Carlo algorithms used in inference for crossed random effects models in modern analysis of variance. We consider a plain Gibbs sampler and propose a simple modification, referred to as a collapsed Gibbs sampler. Under some balancedness conditions on the data designs and assuming that precision hyperparameters are known, we demonstrate that the plain Gibbs sampler is not scalable, in the sense that its complexity is worse than proportional to the number of parameters and data, but the collapsed Gibbs sampler is scalable. In simulated and real datasets we show that the explicit convergence rates predicted by our theory closely match the computable, but nonexplicit rates in cases where the design assumptions are violated. We also show empirically that the collapsed Gibbs sampler extended to sample precision hyperparameters significantly outperforms alternative state-of-the-art algorithms.

Posterior-based proposals for speeding up Markov chain Monte Carlo

“…A key property of DAGs is that the indices for the latent variables e can be ordered such that each element ξ e is conditionally dependent on only those other elements with lower index ξ e' < e (a property known as topological ordering). 4 Simulation results from sequentially sampling each latent variable ξ e (starting from e = 1 up to e = E) from a set of model-defining univariate probability distributions π(ξ e |ξ e' < e ,θ). Consequently, the latent process likelihood in equation (1.1) can be expressed as…”

“…this cannot be done for the Bernoulli, Poisson or Gamma distributions), hence NCPs cannot be used in examples 5.1 and 5.3 below). More complicated schemes which make use of partial CP/NCP proposals and interweaving different parametrizations [4,16,17,27] are not considered here.…”

“…The focus of this paper is to introduce and explore a new approach that helps alleviate these mixing problems, thus reducing the computational time necessary to generate accurate inference. This approach has practical advantages over existing methodologies that aim to address the same problem [2][3][4].…”

“…Another approach is to develop a non-centred parametrization (NCP) [4,16,17] in which a new set of latent variables ξ' (whose distributions are typically independent of the model parameters θ) are introduced. These new variables are related to the original latent variables through some deterministic function ξ = h(ξ', θ, y).…”

Multilevel Linear Models, Gibbs Samplers and Multigrid Decompositions (with Discussion)

“…Also, our results can be used as a building block to derive computational complexity statements about the Gibbs Sampler in the context of multilevel linear model (see e.g. Papaspiliopoulos et al, 2019 for work in that direction). Note that in the context of conditionally Gaussian models the entire Gaussian mean component could be updated in a single block, thus avoiding convergence issues related to single-site updates.…”