Should we be concerned about multiple comparisons in hierarchical Bayesian models?

Abstract

Ecologists increasingly use hierarchical Bayesian (HB) models to estimate group‐level parameters that vary by, for example, species, treatment level, habitat type or other factors. Group‐level parameters may be compared to infer differences among levels. We would conclude a non‐zero pairwise difference, separately, for each pair in the group, when the respective 95% credible interval excludes zero. Classical procedures suggest that the rejection procedure should be adjusted to control the family‐wise error rate (FWER) for a family of differences. Adjustments for FWER have been considered unnecessary in HB models due to partial pooling whereby increased pooling strength – group‐level parameters become more alike – could lead to decreased rejection rates (Type I error, FWER, or Power) and increased false acceptance rates (Type 2 error and its family‐wise analogue). To address this, we conducted a simulation experiment with factors of sample size, group size, balance (missingness), overall mean and ratio of within‐ to between‐group variances, resulting in 2016 factor‐level combinations (‘scenarios’), replicated 100 times, producing 201,600 pseudo datasets analysed in a Bayesian framework. We evaluated the results in the context of a new partial pooling index (PPI), which we show is also applicable to more complex model structures based on four real‐data examples. Simulation results confirm intuition that rejection rates (false and true) decrease and false acceptance rates increase with increasing PPI or pooling strength (scenario‐level R2 = 0.81–0.97). The relationship with PPI differed greatly for balanced versus unbalanced designs and was affected by group size, especially for family‐wise errors. Critically, an HB model does not guarantee that the FWER will follow a set significance level (α); for example, even minor imbalance can lead to FWER > α for weak to moderate pooling. These results are confirmed by the real‐data examples, suggesting that ecologists need to consider FWER when applying HB models, especially for large group sizes or incomplete datasets. Contrary to current thought, HB models are not immune to issues of multiplicity, and our proposed PPI offers a method for evaluating if a particular HB analysis is likely to produce FWER ≤ α (no adjustment or alternative solution required).

The importance of monsoon precipitation for foundation tree species across the semiarid Southwestern U.S.

“…Likewise, we modeled each q Local s as coming from an overall distribution with parameters that varied at the level of species (site-species combinations are nested in species), and the species-level parameters were given relatively noninformative priors. These hierarchical priors allowed for borrowing of strength (Ogle et al, 2018) among trees within a site (for q) and for sites within each species (q Local ); they also allowed us to compute the expected contribution of soil water from different depths for each species-site combination (e.g., Eq t s,d ), and the expected importance of local soil water for different species (e.g., Eqs Local ).…”

BISoN: A Bayesian Framework for Inference of Social Networks

“…There are several ways to gain inference from these values. The overlap of the credible interval with zero is often used to indicate the presence of a 'significant effect', though this approach has been criticised for several reasons (Gelman et al, 2012;Ogle et al, 2019). An alternative option is to use Bayes factors to quantify support for or against the hypothesis H 1 : β 1 = 0 (Kass and Raftery, 1995;Jeffreys, 1998).…”

Ensuring identifiability in hierarchical mixed effects Bayesian models

“…In particular, the observed units (e.g., trees) are typically treated as conditionally independent, arising from a common probability distribution described by (conditional on), for example, variance and/or covariance parameters that quantify variability among the units. The assumption of exchangeability and a common distribution often results in shrinkage (Gelman and Hill 2007, Qian et al 2010, Ogle et al 2019) of a group of random effects (toward some value, often zero) or, equivalently, partial pooling or borrowing of strength (Gelman and Hill 2007, Carlin and Louis 2009, Qian et al 2010, Ogle et al 2019) among a group of effects (e.g., among individual trees, the group levels or units, within the plot). And, the degree of partial pooling among units is related to among unit (within group) variability, with smaller variances allowing for the possibility of stronger pooling (Gelman and Hill 2007, Ogle et al 2019).…”

“…The assumption of exchangeability and a common distribution often results in shrinkage (Gelman and Hill 2007, Qian et al 2010, Ogle et al 2019) of a group of random effects (toward some value, often zero) or, equivalently, partial pooling or borrowing of strength (Gelman and Hill 2007, Carlin and Louis 2009, Qian et al 2010, Ogle et al 2019) among a group of effects (e.g., among individual trees, the group levels or units, within the plot). And, the degree of partial pooling among units is related to among unit (within group) variability, with smaller variances allowing for the possibility of stronger pooling (Gelman and Hill 2007, Ogle et al 2019). The exchangeability assumption allows for inference about individuals (units or levels; e.g., individual trees) via individual‐specific (random) effects or about the population from which they came (e.g., the forest represented by the plot) via a common distribution’s variance or covariance parameters (e.g., the within‐group variance terms).…”