Risk of bias: a simulation study of power to detect study-level moderator effects in meta-analysis

Abstract

BackgroundThere are both theoretical and empirical reasons to believe that design and execution factors are associated with bias in controlled trials. Statistically significant moderator effects, such as the effect of trial quality on treatment effect sizes, are rarely detected in individual meta-analyses, and evidence from meta-epidemiological datasets is inconsistent. The reasons for the disconnect between theory and empirical observation are unclear. The study objective was to explore the power to detect study level moderator effects in meta-analyses.MethodsWe generated meta-analyses using Monte-Carlo simulations and investigated the effect of number of trials, trial sample size, moderator effect size, heterogeneity, and moderator distribution on power to detect moderator effects. The simulations provide a reference guide for investigators to estimate power when planning meta-regressions.ResultsThe power to detect moderator effects in meta-analyses, for example, effects of study quality on effect sizes, is largely determined by the degree of residual heterogeneity present in the dataset (noise not explained by the moderator). Larger trial sample sizes increase power only when residual heterogeneity is low. A large number of trials or low residual heterogeneity are necessary to detect effects. When the proportion of the moderator is not equal (for example, 25% ‘high quality’, 75% ‘low quality’ trials), power of 80% was rarely achieved in investigated scenarios. Application to an empirical meta-epidemiological dataset with substantial heterogeneity (I2 = 92%, τ2 = 0.285) estimated >200 trials are needed for a power of 80% to show a statistically significant result, even for a substantial moderator effect (0.2), and the number of trials with the less common feature (for example, few ‘high quality’ studies) affects power extensively.ConclusionsAlthough study characteristics, such as trial quality, may explain some proportion of heterogeneity across study results in meta-analyses, residual heterogeneity is a crucial factor in determining when associations between moderator variables and effect sizes can be statistically detected. Detecting moderator effects requires more powerful analyses than are employed in most published investigations; hence negative findings should not be considered evidence of a lack of effect, and investigations are not hypothesis-proving unless power calculations show sufficient ability to detect effects.

The effect of funding sources on donepezil randomised controlled trial outcome: a meta-analysis

“…This is likely to be an issue in the current meta-analyses, as the samples included contain 10 or fewer studies. This possibility is supported by a recent review which simulated the power of moderator analyses given the number of studies in a meta-analysis, the number of participants in these studies and the expected moderator effect size 25. With reference to their findings, a meta-analysis of the current study's size may not have sufficient power (0.8) to reliably detect a small-to-medium moderator effect.…”

Is Participatory Design Associated with the Effectiveness of Serious Digital Games for Healthy Lifestyle Promotion? A Meta-Analysis

“…A sufficient sample was defined as having at least three studies in a category, or a minimum sample size of 250 participants per category. A sample size smaller than this was shown not to have sufficient power to detect even large, homogeneous moderator effects [46]. This minimum sample size was thus chosen to make meaningful moderator analyses, without being overly restrictive.…”

How Effective Are Active Videogames Among the Young and the Old? Adding Meta-analyses to Two Recent Systematic Reviews

“…It should be noted, however, that the statistical power of moderator analyses in meta-analysis is not always high 40 and that a large number of studies is generally needed to detect effects. 41 Given that research about active videogames is still in its infancy, so that not many published studies exist, 9,16 the results of these analyses should be considered with caution. No moderator analyses were performed in case of limited heterogeneity.…”

“…32 In addition, moderator analyses have only limited power with a small set of studies. 40,41 This latter limitation is particularly noteworthy given the substantial heterogeneity that was found in the elderly studies. Our results revealed that 68 percent of the variance in effect sizes across the elderly studies was attributable to systematic differences.…”

“…In future, meta-analytic procedures remain necessary to obtain valid estimates of active videogame effects. With additional empirical studies coming forth, effect size estimates are likely to become more reliable, moderator analyses will have increased power, 40,41 and it will be possible to test for publication bias. 42 …”