Methodological problems in the use of indirect comparisons for evaluating healthcare interventions: survey of published systematic reviews

Abstract

Objective To investigate basic assumptions and other methodological problems in the application of indirect comparison in systematic reviews of competing healthcare interventions.Design Survey of published systematic reviews.Inclusion criteria Systematic reviews published between 2000 and 2007 in which an indirect approach had been explicitly used. Identified reviews were assessed for comprehensiveness of the literature search, method for indirect comparison, and whether assumptions about similarity and consistency were explicitly mentioned.Results The survey included 88 review reports. In 13 reviews, indirect comparison was informal. Results from different trials were naively compared without using a common control in six reviews. Adjusted indirect comparison was usually done using classic frequentist methods (n=49) or more complex methods (n=18). The key assumption of trial similarity was explicitly mentioned in only 40 of the 88 reviews. The consistency assumption was not explicit in most cases where direct and indirect evidence were compared or combined (18/30). Evidence from head to head comparison trials was not systematically searched for or not included in nine cases.Conclusions Identified methodological problems were an unclear understanding of underlying assumptions, inappropriate search and selection of relevant trials, use of inappropriate or flawed methods, lack of objective and validated methods to assess or improve trial similarity, and inadequate comparison or inappropriate combination of direct and indirect evidence. Adequate understanding of basic assumptions underlying indirect and mixed treatment comparison is crucial to resolve these methodological problems.

Preventive Pharmacologic Treatments for Episodic Migraine in Adults

“…[60][61][62][63][64] First, we used adjusted indirect frequentist comparisons for individual drugs compared with placebo. 62 This analysis provided pairwise triangular comparisons for drugs compared with placebo rather than network meta-analysis.…”

“…Second, to address the problems with inevitable differences across studies, we used mixed (or multiple) treatment comparison (MTCs) Bayesian network meta-analysis. [62][63][64] We calculated Bayesian odds ratios 43,51 with 2.5 to 97.5 % credible intervals and Bayesian network random effects metaanalysis assuming heterogeneous variances across treatments (online Appendix Table 3). 65 We synthesized evidence from drug classes in network meta-analysis when individual drugs from the same class demonstrated no significant differences in outcomes.…”

Efficacy of supplemental peripheral nerve blockade for hip fracture surgery: multiple treatment comparison

“…While this method yields robust data about the absolute performance of each treatment, it is not methodologically valid simply to collect the aggregate effect sizes of each block and determine the best treatment by choosing the biggest number, because this approach would ignore any data we have from direct comparisons. 5 How then can we determine best practice? The difficulty of performing an omnibus multi-armed RCT to find the single best option becomes obvious if, for the sake of discussion, we accept that there are (at least) seven different anatomic approaches to the nerves innervating the hip, three different technical methods of nerve location, catheter vs single-shot options, and several different local anesthetics that can be administered in any number of concentrations and doses, with many different additives, and for a variety of durations.…”

Appropriate use of information in therapeutic decision-making: reflections on indirect comparisons

“…This is particularly important, as although obviously fundamentally flawed analyses are now far less commonly seen in the literature than was the case five years ago 14 , more subtle errors that may slip past the unwary reader are still prevalent.…”

“…It is beyond the scope of this editorial to tease apart the strengths and weaknesses of the different approaches to indirect comparisons -for those who are interested, there are a number of useful publications that can help with this 6,7,9,13,14 . What is important to grasp is that each indirect evidence set has its own individual challenges, with several potential approaches to meaningful analysis being available.…”