Evaluation of the Use of Cancer Registry Data for Comparative Effectiveness Research

Abstract

IMPORTANCEResearchers often analyze cancer registry data to assess for differences in survival among cancer treatments. However, the retrospective, nonrandomized design of these analyses raises questions about study validity.OBJECTIVE To examine the extent to which comparative effectiveness analyses using observational cancer registry data produce results concordant with those of randomized clinical trials. DESIGN, SETTING, AND PARTICIPANTSIn this comparative effectiveness study, a total of 141 randomized clinical trials referenced in the National Comprehensive Cancer Network Clinical Practice Guidelines for 8 common solid tumor types were identified. Data on participants within the National Cancer Database (NCDB) diagnosed between 2004 and 2014, matching the eligibility criteria of the randomized clinical trial, were obtained. The present study was conducted from August 1, 2017, to September 10, 2019. The trials included 85 118 patients, and the corresponding NCDB analyses included 1 344 536 patients. Three Cox proportional hazards regression models were used to determine hazard ratios (HRs) for overall survival, including univariable, multivariable, and propensity score-adjusted models. Multivariable and propensity score analyses controlled for potential confounders, including demographic, comorbidity, clinical, treatment, and tumor-related variables. MAIN OUTCOMES AND MEASURESThe main outcome was concordance between the results of randomized clinical trials and observational cancer registry data. Hazard ratios with an NCDB analysis were considered concordant if the NDCB HR fell within the 95% CI of the randomized clinical trial HR. An NCDB analysis was considered concordant if both the NCDB and clinical trial P values for survival were nonsignificant (P Ն .05) or if they were both significant (P < .05) with survival favoring the same treatment arm in the NCDB and in the randomized clinical trial. RESULTSAnalyses using the NCDB-produced HRs for survival were concordant with those of 141 randomized clinical trials in 79 univariable analyses (56%), 98 multivariable analyses (70%), and 90 propensity score models (64%). The NCDB analyses produced P values concordant with randomized clinical trials in 58 univariable analyses (41%), 65 multivariable analyses (46%), and 63 propensity score models (45%). No clinical trial characteristics were associated with concordance between NCDB analyses and randomized clinical trials, including disease site, type of clinical intervention, or severity of cancer. CONCLUSIONS AND RELEVANCEThe findings of this study suggest that comparative effectiveness research using cancer registry data often produces survival outcomes discordant with those of (continued)

Assessment of a Clinical Trial–Derived Survival Model in Patients With Metastatic Castration-Resistant Prostate Cancer

“…In the shifting landscape of clinical criterion standards, the increasing use of real-world data is likely. 10 Emerging findings highlight the need to understand the challenges and opportunities of using these data for evidence generation, 31 which is important for future pragmatic approaches toward clinical trials. We present a timely and innovative approach to validate criterion-standard evidence from RCTs using routinely collected clinical data that may augment the implementation of data-driven decisions at the point of care.…”

Bias Reduction through Analysis of Competing Events (BRACE): A Novel Method to Mitigate Bias from Residual Confounding in Observational Data

“…This problem frequently undermines the strength of conclusions from observational studies and can result in erroneous inferences. [1][2][3][4][5][6][7][8][9] When effects of treatments on competing events can be bounded a priori, however, residual confounding can often be diagnosed using competing risks analysis. 9 Building on a proportional relative hazards model, we derived a method to mitigate bias due to residual confounding when it is identified, resulting in lower model error compared to standard approaches.…”

“…3 However, residual confounding from unknown and unmeasured confounders is still a pernicious problem that can undermine conclusions from such analyses and cannot be overcome by scoring and weighting methods. [4][5][6][7][8] Competing event analysis is an underutilized method that allows for the identification of residual confounding problems in non-randomized data sources, particularly when the effect of a treatment on competing events can be bounded a priori. 9 For example, while the simple addition of a novel cancer treatment to a standard regimen may have no effect on or even increase mortality from non-cancer health events, such as cardiac disease, it typically would not reduce the incidence of such events.…”

Prevalence of missing data in the National Cancer Database and association with overall survival

“…36 Recent systematic comparisons of registry studies and randomized trials do not demonstrate concordant results. 10,12 Poor quality documentation is therefore a major obstacle to modern RWD sources and can introduce significant biases when measuring survival outcomes. We demonstrate a consistent association of missing data with worse survival across multiple cancer types, particularly amongst patients with advanced cancers.…”

“…This is particularly germane given emerging evidence suggest that treatment-associated survival outcomes using registry and similar randomized controlled trials are not concordant. 10-12 As the reliance on clinical registries for evidence generation is likely to grow, there is critical need to assess the quality of clinical evidence generated from registry and other RWD sources, as well as their adherence to best data practices.…”