Methods for the Inclusion of Real-World Evidence in a Rare Events Meta-Analysis of Randomized Controlled Trials

Abstract

Background: Many rare events meta-analyses of randomized controlled trials (RCTs) have lower statistical power, and real-world evidence (RWE) is becoming widely recognized as a valuable source of evidence. The purpose of this study is to investigate methods for including RWE in a rare events meta-analysis of RCTs and the impact on the level of uncertainty around the estimates. Methods: Four methods for the inclusion of RWE in evidence synthesis were investigated by applying them to two previously published rare events meta-analyses: the naïve data synthesis (NDS), the design-adjusted synthesis (DAS), the use of RWE as prior information (RPI), and the three-level hierarchical models (THMs). We gauged the effect of the inclusion of RWE by varying the degree of confidence placed in RWE. Results: This study showed that the inclusion of RWE in a rare events meta-analysis of RCTs could increase the precision of the estimates, but this depended on the method of inclusion and the level of confidence placed in RWE. NDS cannot consider the bias of RWE, and its results may be misleading. DAS resulted in stable estimates for the two examples, regardless of whether we placed high- or low-level confidence in RWE. The results of the RPI approach were sensitive to the confidence level placed in RWE. The THM was effective in allowing for accommodating differences between study types, while it had a conservative result compared with other methods. Conclusion: The inclusion of RWE in a rare events meta-analysis of RCTs could increase the level of certainty of the estimates and enhance the decision-making process. DAS might be appropriate for inclusion of RWE in a rare event meta-analysis of RCTs, but further evaluation in different scenarios of empirical or simulation studies is still warranted.

The Disruptive Force of Real-World Evidence

“…Additionally, when investigating safety, RCTs and RWE can complement each other. This has recently been exemplified by Yao et al [ 6 ], who found that including RWE in a rare events meta-analysis had the potential to corroborate findings from RCTs, increase precision, and consequently enhance the decision-making process.…”

Evaluating the impact of including non-randomised studies of interventions in meta-analysis of randomised controlled trials: a protocol for a meta-epidemiological study

“…However, they used random and common effects models to quantitatively synthesise RCTs and cohort studies; these methods do not attempt to adjust for potential bias from cohort studies. 26 No prior review has empirically investigated the influence on estimates of pooled BoE integrated from RCTs and NRSIs using advanced statistical methods, such as a bias-corrected meta-analysis model. 27 These shortcomings leave important questions unanswered, which we will address with three main objectives in our meta-epidemiological study.…”

“…In a study of 118 BoE paired based on RCTs and cohort studies reported in 13 high impact factor medical journals, Bröckelmann et al 25 investigated whether inclusion of evidence from cohort studies modified the conclusion when only evidence from RCTs was considered. However, they used random and common effects models to quantitatively synthesise RCTs and cohort studies; these methods do not attempt to adjust for potential bias from cohort studies 26. No prior review has empirically investigated the influence on estimates of pooled BoE integrated from RCTs and NRSIs using advanced statistical methods, such as a bias-corrected meta-analysis model 27…”