Concordance of Clinician-Documented and Imaging Response in Patients With Stage IV Non–Small Cell Lung Cancer Treated With First-Line Therapy

Abstract

IMPORTANCEIn observational oncology studies of solid tumors, response to treatment can be evaluated based on electronic health record (EHR) documentation (clinician-assessed response [CAR]), an approach different from standardized radiologist-measured response (Response Evaluation Criteria in Solid Tumours [RECIST] 1.1). OBJECTIVE To evaluate the feasibility of an imaging response based on RECIST (IRb-RECIST) and the concordance between CAR and imaging response based on RECIST assessments, and investigate discordance causes. DESIGN, SETTING, AND PARTICIPANTS This cohort study used an EHR-derived, deidentified database that included patients with stage IV non-small cell lung cancer (NSCLC) diagnosed between January 1, 2011, to June 30, 2019, selected from 3 study sites. Data analysis was conducted in August, 2020.EXPOSURES Undergoing first-line therapy and imaging assessments of response to treatment. MAIN OUTCOMES AND MEASURESIn this study, CAR assessments (referred to in prior publications as "real-world response" [rwR]) were defined as clinician-documented changes in disease burden at radiologic evaluation time points; they were abstracted manually and assigned to response categories. The RECIST-based assessments accommodated routine practice patterns by using a modified version of RECIST 1.1 (IRb-RECIST), with independent radiology reads. Concordance was calculated as the percent agreement across all response categories and across a dichotomous stratification (response [complete or partial] vs no response), unconfirmed or confirmed. RESULTSThis study found that, in 100 patients evaluated for concordance, agreement between CAR and IRb-RECIST was 71% (95% CI, 61%-80%), and 74% (95% CI, 64%-82%) for confirmed and unconfirmed response, respectively. There were more responders using CAR than IRb-RECIST (40 vs 29 with confirmation; 64 vs 43 without confirmation). The main sources of discordance were the different use of thresholds for tumor size changes by RECIST vs routine care, and unavailable baseline or follow-up scans resulting in inconsistent anatomic coverage over time. CONCLUSIONS AND RELEVANCEIn this cohort study of patients with stage IV NSCLC, we collected routine-care imaging, showing the feasibility of response evaluation using IRb-RECIST criteria with independent centralized review. Concordance between CAR and centralized IRb-RECIST was moderate. Future work is needed to evaluate the generalizability of these results to broader populations, and investigate concordance in other clinical settings.

Raising the Bar for Real-World Data in Oncology: Approaches to Quality Across Multiple Dimensions

“… 36 In another case, Flatiron Health created an external reference standard of radiologist-measured response via RECIST 1.1 to validate clinician-assessed response on the basis of EHR documentation. 35 Although robust, such validations are often labor-intensive or infeasible.…”

“…34 Complex clinical concepts captured from unstructured data typically require a more robust validation process than variables that are structured data at the source. [35][36][37] For example, variables such as birth year or sex are commonly entered into structured fields to support clinical workflows and are expected to be verified at multiple points during care. Although verification of data processing and pipelines remains essential to data integrity, validation of such data against external references is less critical, given that the value is expected to be accurate and complete to support clinical procedures.…”

Comment on “Evaluation of pulmonary nodules by magnetic resonance imaging sequences: which sequence will replace computed tomography?”

“…Cancer therapy is based on histology combined with all imaging findings and the presence of local and distant metastases that should be regularly followed up for the progression of the disease. Screening in those cases assesses the need for a second or third line of therapy 7 . The restaging should be based on RECIST 1.1 criteria 8 .…”

The Bar Is High: Evaluating Fit-for-Use Oncology Real-World Data for Regulatory Decision Making

“…11 This may lead to the potential for misclassification of the outcome and inability to properly estimate the effect size, as physician-reported response rate may overestimate response rate according to RECIST criteria in the real-world setting. 12,13 An externally controlled trial may underestimate the effect of the medical product if physician-reported response is used in the RWD-derived external control arm. Likewise, a postmarketing study may be interested in characterizing medical product effectiveness in patients who were underrepresented in the pivotal clinical trial, such as older adults.…”