Putting Variation into Variance: Modeling Between-Study Heterogeneity in Meta-Analysis

Abstract

We shed much needed light upon a critical assumption that is oft-overlooked---or not considered at all---in random-effects meta-analysis.Namely, that between-study variance is constant across \emph{all} studies which implies they are from the \emph{same} population. Yet it is not hard to imagine a situation where there are several and not merely one population of studies, perhaps differing in their between-study variance (i.e., heteroskedasticity). The objective is to then make inference, given that there are variations in heterogeneity. There is an immediate problem, however, in that modeling heterogeneous variance components is not straightforward to do in a general way. To this end, we propose novel methodology, termed Bayesian location-scale meta-analysis, that can accommodate moderators for both the overall effect (location) and the between-study variance (scale). After introducing the model, we then extend heterogeneity statistics, prediction intervals, and hierarchical shrinkage, all of which customarily assume constant heterogeneity, to include variations therein. With these new tools in hand, we go to work demonstrating that quite literally \emph{everything} changes when between-study variance is not constant across studies. The changes were not small and easily passed the interocular trauma test---the importance hits right between the eyes. Such examples include (but are not limited to) inference on the overall effect, a compromised predictive distribution, and improper shrinkage of the study-specific effects. Further, we provide an illustrative example where heterogeneity was not considered a mere nuisance to show that modeling variance for its own sake can provide unique inferences, in this case into discrimination across nine countries. The discussion includes several ideas for future research. We have implemented the proposed methodology in the {\tt R} package \textbf{blsmeta}.

Heterogeneous heterogeneity by default: Testing categorical moderators in mixed‐effects meta‐analysis

“…To accomplish the goal of estimating and incorporating heterogeneous variances into tests of moderator effects, we propose using a MELSM (Viechtbauer & López‐López, 2022; Williams et al, 2021). This technique was first introduced outside of meta‐analysis (Hedeker et al, 2008, 2012), and has also been studied under the terms doubly hierarchical model (Lee & Nelder, 2006) and distributional regression (Bürkner, 2018).…”

“…In this sense, the MEM can be considered a special case of the MELSM. Additionally, the coefficients γ 0 and γ 1 are typically estimated using either REML (Viechtbauer, 2021) or Bayesian methods (Williams et al, 2021).…”

“…In this section and the one following it, we focus on frequentist estimation of moderator effects, but it should be mentioned that the same logic we use would similarly hold for Bayesian estimation (Williams et al, 2021). Testing a moderator effect entails computing a series of weighted means.…”

“…Under the MELSM, the weights are given by Williams et al (2021)$$$$ {w}_{ij}^{\ast \ast }=\frac{1}{\sigma_{ij}^2+{\tau}_i^2}. $$$$…”

“…Owing to their flexibility to examine multiple predictors within a single modelling framework, a popular alternative to subgroup analyses are mixed‐effects meta‐regression models (Thompson & Sharp, 1999). Standard mixed‐effects meta‐regressions, or MEMs, do not permit heterogeneous variances between subgroups, but Viechtbauer and López‐López (2022) recently described a mixed‐effects location‐scale model (MELSM) in order to overcome this limitation, while Williams et al (2021) introduced this model under a Bayesian framework. Using a MELSM, separate estimates of τ 2 can be obtained for each level of the moderator (scale), which in turn yield modified estimates of so‐called moderator effects (location).…”