The dark side of the force: Multiplicity issues in network meta‐analysis and how to address them

Efthimiou, Orestis; White, Ian R.

doi:10.1002/jrsm.1377

Cited by 31 publications

(19 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our readers should also note that although we use P‐values to communicate uncertainty, we do not use or promote the concept of “statistical significance,” that is, we have avoided dichotomizing the evidence as significant or not according to an arbitrary threshold, such as P ‐value = .05 or any other value. Statistical significance has been a target for much criticism in science in general, 20 and especially for NMA 21 . We hereby also recommend meta‐analysts to avoid using it to characterize NMA results.…”

Section: Discussionmentioning

confidence: 97%

See 1 more Smart Citation

The Kilim plot: A tool for visualizing network meta‐analysis results for multiple outcomes

Seo

Furukawa

Veroniki

et al. 2020

Research Synthesis Methods

Self Cite

View full text Add to dashboard Cite

Network meta-analysis (NMA) can be used to compare multiple competing treatments for the same disease. In practice, usually a range of outcomes is of interest. As the number of outcomes increases, summarizing results from multiple NMAs becomes a nontrivial task, especially for larger networks. Moreover, NMAs provide results in terms of relative effect measures that can be difficult to interpret and apply in everyday clinical practice, such as the odds ratios. In this article, we aim to facilitate the clinical decision-making process by proposing a new graphical tool, the Kilim plot, for presenting results from NMA on multiple outcomes. Our plot compactly summarizes results on all treatments and all outcomes; it provides information regarding the strength of the statistical evidence of treatment effects, while it illustrates absolute, rather than relative, effects of interventions. Moreover, it can be easily modified to include considerations regarding clinically important effects. To showcase our method, we use data from a network of studies in antidepressants. All analyses are performed in R and we provide the source code needed to produce the Kilim plot, as well as an interactive web application.

show abstract

Section: Discussionmentioning

confidence: 97%

“…Statistical significance has been a target for much criticism in science in general, 20 and especially for NMA. 21 We hereby also recommend meta-analysts to avoid using it to characterize NMA results.…”

Section: Discussionmentioning

confidence: 99%

The Kilim plot: A tool for visualizing network meta‐analysis results for multiple outcomes

Seo

Furukawa

Veroniki

et al. 2020

Research Synthesis Methods

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recent publications have highlighted problems with null hypothesis testing, 51 , 52 particularly in network meta-analysis. 53 Therefore, we did not use the concept of statistical significance when presenting or discussing results from network meta-analyses but instead focused on the clinical interpretation in relation to the corresponding point estimates and their respective confidence intervals.…”

Section: Methodsmentioning

confidence: 99%

Safety and Outcomes of Different Surgical Techniques for Cubital Tunnel Decompression

et al. 2020

View full text Add to dashboard Cite

IMPORTANCE Cubital tunnel syndrome is the second most common compressive neuropathy, affecting 6% of the population. Numerous different operations are performed globally to treat it; however, prior conventional (pairwise) meta-analyses have been unable to determine which procedure is associated with the best outcomes and fewest complications. OBJECTIVE To evaluate which operation for cubital tunnel syndrome is associated with the greatest likelihood of symptomatic cure. DATA SOURCES PubMed, EMBASE, and CENTRAL were searched from database inception to March 2, 2019, with no restrictions on the setting or design of studies. STUDY SELECTION Experimental and observational studies directly comparing the outcomes of at least 2 surgical treatments for adults with primary cubital tunnel syndrome were included. Case reports were excluded, and when comparative studies had subgroups with 1 participant, the singleparticipant subgroup was excluded. The treatments had to be in situ decompression with or without medial epicondylectomy or an anterior subcutaneous, subfascial, intramuscular, or submuscular transposition. The access could be open, minimally invasive, or endoscopic. The comparator could be sham surgery or any operation mentioned earlier. DATA EXTRACTION AND SYNTHESIS Data were extracted by 2 independent reviewers, following the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guideline and the PRISMA Network Meta-analysis extension statement. Network meta-analysis was used to estimate the relative efficacy and safety associated with interventions using relative risks. Surgical techniques were ranked by their probability of being the best (P score) and interpreted in terms of their clinical impact. MAIN OUTCOMES AND MEASURES The primary outcome was response to treatment (ie, symptomatic improvement). The secondary outcomes were perioperative complications, reoperation, and recurrence. RESULTS A total of 30 studies of 2894 limbs undergoing 8 different operations were included. Across the studies, 56% of participants were men, the mean (SD) age was 48 (8) years, and patients had symptoms for a mean (SD) of 15 (7) months. Overall, 87% (95% CI, 92%-91%) of patients improved with surgery; all forms of in situ decompression were more effective than any type of transposition procedure; for example, open in situ decompression with epicondylectomy was (continued) Key Points Question For adults with primary cubital tunnel syndrome, which operation is associated with the best chance of symptomatic cure and lowest risk of complications? Findings This network meta-analysis included 30 studies comparing 8 different operations in 2894 limbs. It found that 87% of patients improve with surgery and that open in situ decompression (with or without a medial epicondylectomy) was associated with the greatest response to treatment and lowest complication risk. Meaning The findings of this study suggest that for adults with primary cubital tunnel syndrome, the most beneficial operation appears to be ope...

show abstract

“…Multiplicity issues were accounted for by using a symmetric random-effects NMA model with exchangeable treatment effects [26], which have been shown to fit well when there is no obvious placebo or other reference treatment in the network, as it was the case in the present analysis.…”

Section: Unadjusted Bayesian Nmamentioning

confidence: 95%

“…Standard NMA models usually do not account for multiple comparisons in estimating relative treatment effects, which might lead to exaggerated and overconfident statements regarding relative treatment effects. The present analysis therefore applied the Bayesian approximation to reduce that problem described by Efthimiou and White [26], where treatment effects are modelled exchangeable, and hence estimates are shrunk away from large values.…”

Section: Limitationsmentioning

confidence: 99%

Comparative efficacy of placebos in short-term antidepressant trials for major depression: a secondary meta-analysis of placebo-controlled trials

Holper

Hengartner

2020

BMC Psychiatry

View full text Add to dashboard Cite

Background The issue of unblinded outcome-assessors and patients has repeatedly been stressed as a flaw in allegedly double-blind antidepressant trials. Unblinding bias can for example result from a drug‘s marked side effects. If such unblinding bias is present for a given drug, then it might be expected that the placebos of that drug are rated significantly less effective than that of other antidepressants. Methods To test this hypothesis, the present exploratory analysis conducted a Bayesian network meta-analysis (NMA) comparing the efficacy of 19 different placebos in placebo-controlled trials provided in the dataset by Cipriani et al. (Lancet 2018; 391: 1357–66). Primary outcome was efficacy (continuous) estimated on the standardized mean difference (SMD) scale and defined as the pre-post change on the Hamilton Depression scale (HAMD-17), on which information was available in N = 258 trials. Results Comparative placebo ranking suggested mirtazapine-placebo (SMD -2.0 [− 5.0–1.0 95% CrI]) to be the most, and amitriptyline- (SMD 1.2 [− 1.6–3.9 95% CrI]) and trazodone- (SMD 2.1 [− 0.9–5.2 95% CrI]) placebos to be the least effective placebos. Other placebos suggested to be more effective than amitriptyline- and trazodone-placebos (based on 95% CrIs excluding zero) were citalopram, desvenlafaxine, duloxetine, escitalopram, fluoxetine, sertraline, and venlafaxine placebos. These NMA results were corroborated by the observation that the relative efficacy between drug and placebo was considerably larger for amitriptyline and trazodone than for instance mirtazapine, duloxetine, and venlafaxine, supported by a small and insignificant correlation between drug-efficacy and placebo-efficacy (r = − 0.202, p = 0.408). Discussion The present exploratory NMA indicates that distinguishable side effects of older drugs may unblind outcome-assessors thus resulting in overestimation of the average drug-placebo difference and underrating bias in placebo-arms, particularly for the older antidepressant drugs amitriptyline and trazodone. If confirmed in prospective studies, these findings suggest that efficacy rankings for antidepressants are susceptible to bias and should be considered unreliable or misleading. The analysis is limited by the focus on the single-comparison placebos (76%, i.e., placebos assessed in two-arm trials), since double-comparison placebos (25%, i.e., placebos assessed in three-arm trials) are hard to interpret and therefore not included in the present interpretation. Another limitation is the problem of multiplicity, which was only approximately accounted for in the Bayesian NMA by modelling treatment effects as exchangeable.

show abstract

The dark side of the force: Multiplicity issues in network meta‐analysis and how to address them

Cited by 31 publications

References 34 publications

The Kilim plot: A tool for visualizing network meta‐analysis results for multiple outcomes

The Kilim plot: A tool for visualizing network meta‐analysis results for multiple outcomes

Safety and Outcomes of Different Surgical Techniques for Cubital Tunnel Decompression

Comparative efficacy of placebos in short-term antidepressant trials for major depression: a secondary meta-analysis of placebo-controlled trials

Contact Info

Product

Resources

About