Predicting the extent of heterogeneity in meta-analysis, using empirical data from the Cochrane Database of Systematic Reviews

Abstract

Background Many meta-analyses contain only a small number of studies, which makes it difficult to estimate the extent of between-study heterogeneity. Bayesian meta-analysis allows incorporation of external evidence on heterogeneity, and offers advantages over conventional random-effects meta-analysis. To assist in this, we provide empirical evidence on the likely extent of heterogeneity in particular areas of health care.Methods Our analyses included 14 886 meta-analyses from the Cochrane Database of Systematic Reviews. We classified each meta-analysis according to the type of outcome, type of intervention comparison and medical specialty. By modelling the study data from all meta-analyses simultaneously, using the log odds ratio scale, we investigated the impact of meta-analysis characteristics on the underlying between-study heterogeneity variance. Predictive distributions were obtained for the heterogeneity expected in future meta-analyses.Results Between-study heterogeneity variances for meta-analyses in which the outcome was all-cause mortality were found to be on average 17% (95% CI 10–26) of variances for other outcomes. In meta-analyses comparing two active pharmacological interventions, heterogeneity was on average 75% (95% CI 58–95) of variances for non-pharmacological interventions. Meta-analysis size was found to have only a small effect on heterogeneity. Predictive distributions are presented for nine different settings, defined by type of outcome and type of intervention comparison. For example, for a planned meta-analysis comparing a pharmacological intervention against placebo or control with a subjectively measured outcome, the predictive distribution for heterogeneity is a log-normal (−2.13, 1.582) distribution, which has a median value of 0.12. In an example of meta-analysis of six studies, incorporating external evidence led to a smaller heterogeneity estimate and a narrower confidence interval for the combined intervention effect.Conclusions Meta-analysis characteristics were strongly associated with the degree of between-study heterogeneity, and predictive distributions for heterogeneity differed substantially across settings. The informative priors provided will be very beneficial in future meta-analyses including few studies.

Estimates of Between-Study Heterogeneity for 705 Meta-Analyses Reported in <i>Psychological Bulletin</i> From 1990–2013

“…In the medical literature, several attempts have been made to present an overview of between-study heterogeneity estimates from a large number of metaanalyses and construct an informed prior distribution based on these estimates [1,2,3]. In the field of psychology, an early overview of meta-analyses on the efficacy of psychological, educational, and behavioral treatments is given by [4].…”

Novel oral anticoagulants for the secondary prevention of cerebral ischemia: a network meta-analysis

“…The underlying assumption of transitivity suggests that all pairwise comparisons in the network do not differ with respect to the distribution of effect modifiers [Turner et al 2012]. We presented the results for the comparative efficacy and tolerability by RR estimates and 95% CI.…”

“…We assumed that heterogeneity is the same for all treatment comparisons and the assessment of statistical heterogeneity in the entire network was evaluated in light of its estimated empirical distribution. We considered values from 0.1 to 0.5 to be reasonable, while values 0.5 to 1.0 were considered to represent fairly high and above 1.0 fairly extreme heterogeneity [Turner et al 2012]. We also evaluated the ranking of all primary and secondary outcomes using ranking probabilities; which treatment is the most efficacious regimen, the second best, the third best, and so on [Turner et al 2012].…”

“…We considered values from 0.1 to 0.5 to be reasonable, while values 0.5 to 1.0 were considered to represent fairly high and above 1.0 fairly extreme heterogeneity [Turner et al 2012]. We also evaluated the ranking of all primary and secondary outcomes using ranking probabilities; which treatment is the most efficacious regimen, the second best, the third best, and so on [Turner et al 2012]. To rank the treatments we used the surface under the cumulative ranking (SUCRA) curve.…”

Comparative Efficacy of Bisphosphonates to Prevent Fracture in Men with Osteoporosis: A Systematic Review with Network Meta-Analyses

“…We based direct probability statements on 50,000-simulation iterations to identify the best and most representative data, assuming comparable interstudy variances for all treatment effects for the same outcomes. The assessment of statistical heterogeneity in the entire network was based on the magnitude of the heterogeneity variance parameter (s 2 ) estimated from the network meta-analysis models [23]. Inconsistency was checked if a comparison loop existed [24][25][26].…”