A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis

Abstract

Network meta-analysis (NMA), combining direct and indirect comparisons, is increasingly being used to examine the comparative effectiveness of medical interventions. Minimal guidance exists on how to rate the quality of evidence supporting treatment effect estimates obtained from NMA. We present a four-step approach to rate the quality of evidence in each of the direct, indirect, and NMA estimates based on methods developed by the GRADE working group. Using an example of a published NMA, we show that the quality of evidence supporting NMA estimates varies from high to very low across comparisons, and that quality ratings given to a whole network are uninformative and likely to mislead. Network meta-analysis (NMA) that simultaneously addresses the comparative effectiveness and/or safety of multiple interventions through combining direct and indirect estimates of effect is rapidly gaining popularity and influence. 1-6 Although NMA approaches appear attractive, 6-8 application of their results requires understanding the quality of the evidence. By quality of evidence, we mean the degree of confidence or certainty one can place in estimates of treatment effects.NMA is sufficiently new that terminology differs between authors and continues to evolve. Box 1 presents a glossary of terms used in this article.Rationale for an approach to rate the quality of evidence from NMA Recently, several articles have provided guidance regarding identification of the evidence for a NMA, 9 statistical aspects of conducting NMA, 10-17 and critical appraisal and interpretation of published NMA.18 19 Few of these, however, provide explicit guidance on how to rate the quality of the evidence. Reports of NMAs often describe the risk of bias of trials included in a NMA (such as method of randomisation, concealment of random allocation, masking, etc). [22][23][24] For example, a recent NMA compared the effects of coronary artery bypass grafting, various stents, and medical treatment on mortality, myocardial infarction, and the need for revascularisation among patients with stable coronary artery disease. The authors stated that appropriate methods of concealment of random allocation were reported for 71 trials (71%). 25 Fifty six trials (56%) reported blind adjudication of clinical outcomes, and for 69 trials (69%) data from intention to treat analyses were available. Although such an assessment of risk of bias describes the entire body of evidence (that is, all trials contributing evidence to the NMA), it does not acknowledge that the risk of bias is likely to differ across the comparisons of the network.1 For example, the risk of bias of studies comparing sirolimus eluting stents versus medical treatment may be considerably less than the risk of bias of studies comparing coronary artery bypass grafting with medical treatment. In addition, risk of bias is only one determinant of quality of evidence. Our confidence in effect estimates will, for instance, also decrease if there are large differences in results from study to study (for exampl...

Which Surgical Treatment for Open Tibial Shaft Fractures Results in the Fewest Reoperations? A Network Meta-analysis

“…Two reviewers (CJF, KNV) independently applied GRADE to make direct comparisons using established methods bias [16,18], and to make indirect comparisons using recently suggested methods from the GRADE working group [32]. GRADE guidance includes rating down indirect comparisons when there is suspicion of possible effect modification attributable to differences in patients, optimal use of interventions, or measurement of outcomes across the direct comparisons that informed the indirect comparisons (which we refer to as intransitivity).…”

“…Incoherence typically is attributable to major differences in the trials that make up each comparison (eg, differences between included patients or cointerventions, or major methodologic differences). We dealt with this issue by first identifying the sources of inconsistency (Tables 4 and 5), and second, by using only the highest confidence comparisons as the best estimate of effect, as recommended by the GRADE working group [32]. Finally, functional outcomes were not reported in any of the trials.…”

Network Meta-analysis: Users’ Guide for Surgeons: Part I - Credibility

“…Ideally, for each comparison, the authors present the estimate from the direct comparisons and its associated certainty, the estimate from the indirect comparison and its associated certainty, and the overall NMA estimate and its associated certainty. Methodology from the The Grades of Recommendation, Assessment, Development and Evaluation (GRADE) working group provides a system for making certainty ratings that considers risk of bias, precision, consistency, directness, and publication bias [7,23].…”

Network Meta-analysis: Users’ Guide for Surgeons: Part II - Certainty

“…When incoherence exists, which results one should use as the best estimate (direct, indirect, or network) remains controversial [6,26]. If one body of evidence warrants higher certainty than the other (this will usually but not always be the direct evidence), one can argue strongly that the body of evidence in which there is more certainty represents the best estimate of the true effect [23]. This approach, rather than always using the network estimate as the best estimate, is what the GRADE Working Group suggests.…”

“…A recent publication presented detailed criteria from the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) Working Group for determining certainty in estimates from a NMA [23]. Newer NMAs likely will incorporate these guidelines for readers; however, in the absence of such reporting, authors should provide information addressing key questions that are related to certainty (Table 1) [9,10].…”