Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis

Abstract

BackgroundMeta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question.MethodsWe conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews.ResultsOur search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P< 0.001) and lower precision (P<0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently.ConclusionThough results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question.

Factors underpinning discrepant findings from reviews on the same topic: A systematic analysis of Cochrane and non-Cochrane reviews

“…These findings support earlier assumptions by Useem et al [9], who discussed different search strategies and/or inclusion/exclusion of studies as potential explanations for discrepancies. In our study, only 50% of non-Cochrane reviews reported searching and inclusion of studies irrespective of language and only 46% irrespective of status of publication.…”

“…Accordingly, in our matched-pair analysis of Cochrane and non-Cochrane reviews we identified, for the first time, inconsistencies in pooled effect estimates between meta-analyses examining the role of physical activity interventions for prevention and treatment of major chronic diseases (Hacke & Nunan, 2019). These results are consistent with those of a recent analysis of pharmacological interventions which showed generally poor overlap of included studies between matched pairs of Cochrane and non-Cochrane reviews [9].…”

“…Previous studies similar to ours have note adequately addressed the reasons for discrepancies among a topic-matched pair [9, 10]. In the present study, non-Cochrane reviews were less likely to find the included studies of its Cochrane review match.…”

“…The evidence to date, however, has not adequately addressed the potential methodological factors underpinning observed discrepancies among topic-matched reviews. Useem et al [9] suggested different search strategies and inclusion and exclusion criteria as potential explanations for discrepancies but did not explicit evaluate these or other factors, instead recommending them for future work.…”

‘Spin’ in reports of clinical research

“…Non-Cochrane systematic reviews are twice as likely to have positive conclusion statements as Cochrane reviews 4. Furthermore, non-Cochrane reviews, when matched to an equivalent Cochrane review on the same topic, are more likely to report larger effect sizes with lower precision than the equivalent Cochrane review 5. In both cases, these findings may well reflect the extent to which methodological complexity is ignored or sidestepped in poorer quality reviews.…”