Characteristics and methods of incorporating randomized and nonrandomized evidence in network meta-analyses: a scoping review

Zhang, Kathryn; Arora, Paul; Sati, Neha; Béliveau, Audrey; Troke, Natalie; Veroniki, Areti Angeliki; Rodrigues, Myanca; Zarin, Wasifa; Tricco, Andrea C.

doi:10.1016/j.jclinepi.2019.04.023

Cited by 7 publications

(6 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These are design-adjusted analyses, in which all trials included in the network meta-analysis involve estimates adjusted according to possible bias and overprecision (based on expert opinions); using informative priors, in which meta-analysis of randomized trials is performed based on priors formulated from meta-analyzing observational studies (Bayesian approach); and three categorical models, in which a meta-analysis is performed for each design, and consequently, the overall effect is acquired by synthesizing all design-specific estimates. Although these approaches have been suggested, according to the scoping review by Zhang et al [ 36 ], the vast majority (74%) of network meta-analyses used naïve pooling without specific consideration.…”

Section: Discussionmentioning

confidence: 99%

Stepwise-Hierarchical Pooled Analysis for Synergistic Interpretation of Meta-analyses Involving Randomized and Observational Studies: Methodology Development

Shin¹,

Rim²

2021

J Med Internet Res

View full text Add to dashboard Cite

Background The necessity of including observational studies in meta-analyses has been discussed in the literature, but a synergistic analysis method for combining randomized and observational studies has not been reported. Observational studies differ in validity depending on the degree of the confounders’ influence. Combining interpretations may be challenging, especially if the statistical directions are similar but the magnitude of the pooled results are different between randomized and observational studies (the ”gray zone”). Objective To overcome these hindrances, in this study, we aim to introduce a logical method for clinical interpretation of randomized and observational studies. Methods We designed a stepwise-hierarchical pooled analysis method to analyze both distribution trends and individual pooled results by dividing the included studies into at least three stages (eg, all studies, balanced studies, and randomized studies). Results According to the model, the validity of a hypothesis is mostly based on the pooled results of randomized studies (the highest stage). Ascending patterns in which effect size and statistical significance increase gradually with stage strengthen the validity of the hypothesis; in this case, the effect size of the observational studies is lower than that of the true effect (eg, because of the uncontrolled effect of negative confounders). Descending patterns in which decreasing effect size and statistical significance gradually weaken the validity of the hypothesis suggest that the effect size and statistical significance of the observational studies is larger than the true effect (eg, because of researchers’ bias). Conclusions We recommend using the stepwise-hierarchical pooled analysis approach for meta-analyses involving randomized and observational studies.

show abstract

Section: Discussionmentioning

confidence: 99%

Stepwise-Hierarchical Pooled Analysis for Synergistic Interpretation of Meta-analyses Involving Randomized and Observational Studies: Methodology Development

Shin¹,

Rim²

2021

J Med Internet Res

View full text Add to dashboard Cite

show abstract

“…We do not recommend using NDS as the main method of analysis to combine rare events data. A recent scoping review found that the NDS approach was the most frequently used in empirical analyses [ 16 ]. However, NDS ignores differences in study designs and cannot consider the potential bias of RWE studies [ 20 ].…”

Section: Discussionmentioning

confidence: 99%

“…Therefore, RCTs and RWE studies should not be directly integrated without considering the bias of the RWE studies. Generalized evidence synthesis statistical approaches for integrating RCTs and RWE studies can provide not only an integrated result but also a quantitative analysis of the influence of RWE on the integrated evidence under different bias assumptions for the RWE studies, which is more attractive for decision-making [ 16 , 17 , 18 , 19 ]. Relevant methods have been published in the statistical literature but have not been systematically verified or widely used in empirical analyses [ 16 ].…”

Section: Introductionmentioning

confidence: 99%

Methods for the Inclusion of Real-World Evidence in a Rare Events Meta-Analysis of Randomized Controlled Trials

Yao

Wang

Mei

et al. 2023

JCM

View full text Add to dashboard Cite

Background: Many rare events meta-analyses of randomized controlled trials (RCTs) have lower statistical power, and real-world evidence (RWE) is becoming widely recognized as a valuable source of evidence. The purpose of this study is to investigate methods for including RWE in a rare events meta-analysis of RCTs and the impact on the level of uncertainty around the estimates. Methods: Four methods for the inclusion of RWE in evidence synthesis were investigated by applying them to two previously published rare events meta-analyses: the naïve data synthesis (NDS), the design-adjusted synthesis (DAS), the use of RWE as prior information (RPI), and the three-level hierarchical models (THMs). We gauged the effect of the inclusion of RWE by varying the degree of confidence placed in RWE. Results: This study showed that the inclusion of RWE in a rare events meta-analysis of RCTs could increase the precision of the estimates, but this depended on the method of inclusion and the level of confidence placed in RWE. NDS cannot consider the bias of RWE, and its results may be misleading. DAS resulted in stable estimates for the two examples, regardless of whether we placed high- or low-level confidence in RWE. The results of the RPI approach were sensitive to the confidence level placed in RWE. The THM was effective in allowing for accommodating differences between study types, while it had a conservative result compared with other methods. Conclusion: The inclusion of RWE in a rare events meta-analysis of RCTs could increase the level of certainty of the estimates and enhance the decision-making process. DAS might be appropriate for inclusion of RWE in a rare event meta-analysis of RCTs, but further evaluation in different scenarios of empirical or simulation studies is still warranted.

show abstract

“…60 Incorporating NRS evidence into NMA models that traditionally only include RCTs is increasingly important in several clinical research settings, such as when conducting RCTs are less feasible for rare conditions. A recent scoping review of methods that combine RCT and NRS in NMA 61 reveals that unadjusted synthesis is the most popular approach, probably for its ease of use. The unadjusted analysis, however, can be considered as an initial step but not the primary analysis, as it ignores the differences in design and RoB.…”

Section: Discussionmentioning

confidence: 99%

Synthesizing cross‐design evidence and cross‐format data using network meta‐regression

Hamza

Chalkou

Pellegrini³

et al. 2023

Research Synthesis Methods

View full text Add to dashboard Cite

In network meta-analysis (NMA), we synthesize all relevant evidence about health outcomes with competing treatments. The evidence may come from randomized clinical trials (RCT) or non-randomized studies (NRS) as individual participant data (IPD) or as aggregate data (AD). We present a suite of Bayesian NMA and network meta-regression (NMR) models allowing for cross-design and cross-format synthesis. The models integrate a three-level hierarchical model for synthesizing IPD and AD into four approaches. The four approaches account for differences in the design and risk of bias (RoB) in the RCT and NRS evidence. These four approaches variously ignoring differences in RoB, using NRS to construct penalized treatment effect priors and bias-adjustment models that control the contribution of information from high RoB studies in two different ways. We illustrate the methods in a network of three pharmacological interventions and placebo for patients with relapsingremitting multiple sclerosis. The estimated relative treatment effects do not change much when we accounted for differences in design and RoB. Conducting network meta-regression showed that intervention efficacy decreases with increasing participant age. We also re-analysed a network of 431 RCT comparing 21 antidepressants, and we did not observe material changes in intervention efficacy when adjusting for studies' high RoB. We re-analysed both case studies accounting for different study RoB. In summary, the described suite of NMA/NMR models enables the inclusion of all relevant evidence while incorporating information on the within-study bias in both observational and experimental data and enabling estimation of individualized treatment effects through the inclusion of participant characteristics.

show abstract

Characteristics and methods of incorporating randomized and nonrandomized evidence in network meta-analyses: a scoping review

Cited by 7 publications

References 40 publications

Stepwise-Hierarchical Pooled Analysis for Synergistic Interpretation of Meta-analyses Involving Randomized and Observational Studies: Methodology Development

Stepwise-Hierarchical Pooled Analysis for Synergistic Interpretation of Meta-analyses Involving Randomized and Observational Studies: Methodology Development

Methods for the Inclusion of Real-World Evidence in a Rare Events Meta-Analysis of Randomized Controlled Trials

Synthesizing cross‐design evidence and cross‐format data using network meta‐regression

Contact Info

Product

Resources

About