Comparing results of large clinical trials to those of meta‐analyses

Abstract

We consider methods for assessing agreement or disagreement between the results of a meta-analysis of small studies addressing a clinical question and the result of a large clinical trial (LCT) addressing the same clinical question. We recommend basing conclusions about agreement upon the difference between the two results (relative risk, log-odds ratio or similar summary statistic), in the light of the estimated standard error of that difference. To estimate the standard error of the meta-analytic result we recommend a random effects analysis, and where a between-studies variance component is found, that component of variance should be used twice: once in the estimated standard error for the meta-analytic result and again in the standard error of the LCT result (augmenting the internal standard error of that statistic). Such broadening of the standard error reduces the appearance of disagreement. We also offer a critique of a different published approach, which is based on consistency of findings of statistical significance, a matter of how the two results regard zero, which is a poor measure of how closely they agree with each other.

RETRACTED: Effect of intranasal oxytocin administration on psychiatric symptoms: A meta-analysis of placebo-controlled studies

“…All effect sizes were within 2 standard deviations of the mean effect size. Finally, the random-effects model, rather than the fixed-effects model, was chosen to calculate effect size estimates given the heterogeneity of the studies (Hedges and Vevea, 1998; Moses et al, 2002). …”

Simulation of the Syst-Eur randomized control trial using a primary care electronic medical record was feasible

“…Differences between the IRRs for the GPRD study and RCT were compared statistically using a standard normal z-test of the difference between the natural logarithms of the IRRs divided by the square root of the variance of that difference [6,38]:…”

Efficacy of Biofeedback in Chronic back Pain: a Meta-Analysis

“…No studies were determined to be outliers using this criterion. Effect size estimates were calculated using a random-effects model rather than a fixed-effects model because the studies were not functionally identical [36,37]. Effect size estimates for follow-up data were also calculated in the manner described above.…”