Between-trial heterogeneity in meta-analyses may be partially explained by reported design characteristics

Abstract

ObjectiveWe investigated the associations between risk of bias judgments from Cochrane reviews for sequence generation, allocation concealment and blinding, and between-trial heterogeneity.Study Design and SettingBayesian hierarchical models were fitted to binary data from 117 meta-analyses, to estimate the ratio λ by which heterogeneity changes for trials at high/unclear risk of bias compared with trials at low risk of bias. We estimated the proportion of between-trial heterogeneity in each meta-analysis that could be explained by the bias associated with specific design characteristics.ResultsUnivariable analyses showed that heterogeneity variances were, on average, increased among trials at high/unclear risk of bias for sequence generation (λ̂ 1.14, 95% interval: 0.57–2.30) and blinding (λ̂ 1.74, 95% interval: 0.85–3.47). Trials at high/unclear risk of bias for allocation concealment were on average less heterogeneous (λ̂ 0.75, 95% interval: 0.35–1.61). Multivariable analyses showed that a median of 37% (95% interval: 0–71%) heterogeneity variance could be explained by trials at high/unclear risk of bias for sequence generation, allocation concealment, and/or blinding. All 95% intervals for changes in heterogeneity were wide and included the null of no difference.ConclusionOur interpretation of the results is limited by imprecise estimates. There is some indication that between-trial heterogeneity could be partially explained by reported design characteristics, and hence adjustment for bias could potentially improve accuracy of meta-analysis results.

Trends and predictors of biomedical research quality, 1990–2015: a meta-research study

“…Several reasons support the use of at least one high risk of bias assessment as the definition for inadequate methods. Some risk of bias domains might translate into more statistical bias than others, but empirical evidence on the relative importance of the risk of bias domains is limited, and the effect of several versus one high risk assessment on research outcomes is unknown 32 33. The empirical relationship between risk of bias assessments and research outcomes (including actual statistical bias) requires further research.…”

Heterogeneity in Cochrane and non-Cochrane meta-analyses in orthodontics

“…The effect of bias-related reporting characteristics of studies included in meta-analyses such as blinding, random sequence generation, allocation concealment or other reporting related items was not assessed. Although empirical evidence from medical research has shown that between-study heterogeneity may be partially explained by the reporting of characteristics responsible for bias [26], this was beyond the scope of the present work and it may constitute ground for future research. Table 4.…”

One-Year Effectiveness Study of Intravitreously Administered Conbercept® Monotherapy in Diabetic Macular Degeneration: A Systematic Review and Meta-Analysis

“…Additionally, there was some significant heterogeneity between included studies. This discrepancy could have occurred as a result of the heterogeneity of patient characteristics such as age and disease severity, comorbidities, methods for diagnosis and evaluation, treatment doses and interval, and study design features [40][41][42][43].…”