Inconsistency between direct and indirect comparisons of competing interventions: meta-epidemiological study

Abstract

Objective To investigate the agreement between direct and indirect comparisons of competing healthcare interventions.Design Meta-epidemiological study based on sample of meta-analyses of randomised controlled trials. Data sources Cochrane Database of Systematic Reviews and PubMed.Inclusion criteria Systematic reviews that provided sufficient data for both direct comparison and independent indirect comparisons of two interventions on the basis of a common comparator and in which the odds ratio could be used as the outcome statistic.Main outcome measure Inconsistency measured by the difference in the log odds ratio between the direct and indirect methods. ResultsThe study included 112 independent trial networks (including 1552 trials with 478 775 patients in total) that allowed both direct and indirect comparison of two interventions. Indirect comparison had already been explicitly done in only 13 of the 85 Cochrane reviews included. The inconsistency between the direct and indirect comparison was statistically significant in 16 cases (14%, 95% confidence interval 9% to 22%). The statistically significant inconsistency was associated with fewer trials, subjectively assessed outcomes, and statistically significant effects of treatment in either direct or indirect comparisons. Owing to considerable inconsistency, many (14/39) of the statistically significant effects by direct comparison became non-significant when the direct and indirect estimates were combined. ConclusionsSignificant inconsistency between direct and indirect comparisons may be more prevalent than previously observed. Direct and indirect estimates should be combined in mixed treatment comparisons only after adequate assessment of the consistency of the evidence. IntroductionRandomised controlled trials to compare competing interventions are often lacking, and this situation is unlikely to improve in the future because of the inevitable tension between the high cost of clinical trials and the continuing introduction of new treatments.1 2 The dearth of evidence from head to head randomised controlled trials has led to increased use of indirect comparison methods to estimate the comparative effects of treatment. [1][2][3][4] Indirect comparison of competing interventions can be generally defined as a comparison of different treatments for a clinical indication by using data from separate randomised controlled trials, in contrast to direct comparison within randomised controlled trials. Indirect comparison based on a common comparator can preserve certain strengths of randomised allocation of patients for estimating comparative effects of treatment.1 5 6 The term "adjusted indirect comparison" is used to refer to this indirect comparison based on a common intervention (fig 1). 7 Mixed treatment comparison (also known RESEARCHas network meta-analysis or multiple treatment meta-analysis) is a more complex method that combines both indirect and direct estimates simultaneously. [8][9][10] The validity of indirect and mixed treatment comparisons dep...

An assessment of the cost-effectiveness of magnetic resonance, including diffusion-weighted imaging, in patients with transient ischaemic attack and minor stroke: a systematic review, meta-analysis and economic evaluation

“…Observational comparisons may be misleading or uninformative. 179 The principle of using the least biased evidence, which is most likely to come from randomised trials, applies even more to technologies like diagnostic imaging -diagnostic imaging affects treatment decisions and therefore clinical outcomes and should be regarded as no different to the testing of a new drug. The methods required for testing diagnostic imaging may be a bit different but the principle of identifying the benefit and hazard is the same as for drugs or surgical treatments.…”

Bibliographic study showed improving statistical methodology of network meta-analyses published between 1999 and 2015

“…The assessment of risk of bias in the included studies [13][14][15], the magnitude and determinants of heterogeneity [16][17][18], the relative advantages of different methods to evaluate publication bias and small-study effects, [19][20][21] and the importance of a comprehensive search for relevant studies [22] are examples of meta-epidemiological studies that have guided the choice of optimal methods. We are aware of only three such metaepidemiological studies in NMA: Song et al evaluated the prevalence of inconsistency in networks with three treatments [23,24], Veroniki et al studied the prevalence of inconsistency in complex NMAs that included at least four treatments using two alternative methods [25] while Chaimani et al have provided empirical evidence about the impact of risk of bias and small study effects [26].…”

Bayesian Methods Affirm the Use of Percutaneous Coronary Intervention to Improve Survival in Patients With Unprotected Left Main Coronary Artery Disease

“…It is recognized that indirect comparisons performed in a network metaanalysis may give misleading results when trial numbers are small. 32 For this reason, we consider the primary analysis using the 1-year mortality data to be supportive of the existing revascularization guidelines 10,12 and the additional 2-year and 3-year mortality analyses to be sensitivity analyses. …”