Confidence intervals for the between-study variance in random-effects meta-analysis using generalised heterogeneity statistics: should we use unequal tails?

Abstract

BackgroundConfidence intervals for the between study variance are useful in random-effects meta-analyses because they quantify the uncertainty in the corresponding point estimates. Methods for calculating these confidence intervals have been developed that are based on inverting hypothesis tests using generalised heterogeneity statistics. Whilst, under the random effects model, these new methods furnish confidence intervals with the correct coverage, the resulting intervals are usually very wide, making them uninformative.MethodsWe discuss a simple strategy for obtaining 95 % confidence intervals for the between-study variance with a markedly reduced width, whilst retaining the nominal coverage probability. Specifically, we consider the possibility of using methods based on generalised heterogeneity statistics with unequal tail probabilities, where the tail probability used to compute the upper bound is greater than 2.5 %. This idea is assessed using four real examples and a variety of simulation studies. Supporting analytical results are also obtained.ResultsOur results provide evidence that using unequal tail probabilities can result in shorter 95 % confidence intervals for the between-study variance. We also show some further results for a real example that illustrates how shorter confidence intervals for the between-study variance can be useful when performing sensitivity analyses for the average effect, which is usually the parameter of primary interest.ConclusionsWe conclude that using unequal tail probabilities when computing 95 % confidence intervals for the between-study variance, when using methods based on generalised heterogeneity statistics, can result in shorter confidence intervals. We suggest that those who find the case for using unequal tail probabilities convincing should use the ‘1–4 % split’, where greater tail probability is allocated to the upper confidence bound. The ‘width-optimal’ interval that we present deserves further investigation.Electronic supplementary materialThe online version of this article (doi:10.1186/s12874-016-0219-y) contains supplementary material, which is available to authorized users.

Methods to calculate uncertainty in the estimated overall effect size from a random‐effects meta‐analysis

“…When all variance components, including the between‐study variances, are fixed and known, the expected value of Q gen is k − 1, which equals the degrees of freedom of the associated χ 2 distribution . Hence, the small‐sample adjustment q will tend to be close to 1.…”

“…128 When all variance components, including the between-study variances, are fixed and known, the expected value of Q gen is k − 1, which equals the degrees of freedom of the associated χ 2 distribution. [129][130][131] Hence, the small-sample adjustment q will tend to be close to 1. However, q may in fact turn out to be much smaller than 1, such as in cases where the effect sizes are very homogeneous or when the number and/or size of studies is small.…”

Psychological interventions using virtual reality for pain associated with medical procedures: a systematic review and meta-analysis

“…Other important caveats relate to heterogeneity around effect estimates being generally high and even extremely high, or with large confidence intervals (Jackson and Bowden, 2016) across most analyses, and generally not diminished in sensitivity analyses. High heterogeneity impacts the precision of the effect estimates, and raises questions about their reliability (Ioannidis et al ., 2007).…”

Methods for estimating between‐study variance and overall effect in meta‐analysis of odds ratios

“…Viechtbauer and Jackson and Bowden compare CI estimators of the between‐study variance. Interval estimators recommended by Veroniki et al include profile likelihood, the QP interval, and the generalized QP intervals of Biggerstaff and Jackson and Jackson .…”