Automated Electronic Health Record–Based Tool for Identification of Patients With Metastatic Disease to Facilitate Clinical Trial Patient Ascertainment

Abstract

PURPOSE To facilitate identification of clinical trial participation candidates, we developed a machine learning tool that automates the determination of a patient's metastatic status, on the basis of unstructured electronic health record (EHR) data. METHODS This tool scans EHR documents, extracting text snippet features surrounding key words (such as metastatic, progression, and local). A regularized logistic regression model was trained and used to classify patients across five metastatic categories: highly likely and likely positive, highly likely and likely negative, and unknown. Using a real-world oncology database of patients with solid tumors with manually abstracted information as reference, we calculated sensitivity, specificity, negative predictive value (NPV), and positive predictive value (PPV). We validated the performance in a real-world data set, evaluating accuracy gains upon additional user review of tool's outputs after integration into clinic workflows. RESULTS In the training data set (N = 66,532), the model sensitivity and specificity (% [95% CI]) were 82.4 [81.9 to 83.0] and 95.5 [95.3 to 96.7], respectively; the PPV was 89.3 [88.8 to 90.0], and the NPV was 94.0 [93.8 to 94.2]. In the validation sample (n = 200 from five distinct care sites), after user review of model outputs, values increased to 97.1 [85.1 to 99.9] for sensitivity, 98.2 [94.8 to 99.6] for specificity, 91.9 [78.1 to 98.3] for PPV, and 99.4 [96.6 to 100.0] for NPV. The model assigned 163 of 200 patients to the highly likely categories. The error prevalence was 4% before and 2% after user review. CONCLUSION This tool infers metastatic status from unstructured EHR data with high accuracy and high confidence in more than 75% of cases, without requiring additional manual review. By enabling efficient characterization of metastatic status, this tool could mitigate a key barrier for patient ascertainment and clinical trial participation in community clinics.

Utilization of EHRs for clinical trials: a systematic review

“… Natalie C. Ernecof 2018 [ 22 ] USA To develop an EHR phenotype for identifying patients with late-stage dementia for a clinical trial of palliative care consultation. Jae Hyun Kim 2021 [ 23 ] USA To evaluate the impact of eligibility criteria on recruitment and observable clinical outcomes of COVID-19 clinical trials using EHR data Jeffrey Kirshner 2021 [ 24 ] USA To facilitate identification of clinical trial participation candidates, the researchers developed a machine learning tool that automates the determination of a patient's metastatic status, on the basis of unstructured EHR data. Niina Laaksonen 2021 [ 25 ] Finland To evaluate the accuracy of a commercially available EHR Research Platform, “InSite”, in identifying potential trial participants from the EHR system of a large tertiary care hospital.…”

“…This method promises to improve feasibility and efficiency for COVID-19 clinical trial recruitment. Jeffrey Kirshner 2021 [ 24 ] This tool infers from unstructured EHR data with high accuracy and high confidence in more than 75% of cases, without requiring additional manual review. This tool could mitigate a key barrier for patient ascertainment and clinical trial participation in community clinics.…”

Use of Natural Language Processing to Infer Sites of Metastatic Disease From Radiology Reports at Scale

“…32,33 Finally, these models could be used to provide relevant information for identification of potential patients for clinical trials (eg, presence of brain or visceral metastases). However, clinical trial matching is complex, 34,35 and NLP may just be one component of a successful solution. 36-38…”

Artificial intelligence to unlock real‐world evidence in clinical oncology: A primer on recent advances

“… 83 This strategy led to an increase in serious illness conversations and reduction in end‐of‐life systemic therapy administration. 83 ML tools have been developed to identify patients eligible for enrollment on clinical trials of new cancer therapies 84 , 85 and to explore inefficiencies in restrictive clinical trial enrollment criteria relative to real‐world populations. 86 Despite these efforts, prospective validation of ML models and rigorous testing of their impact on clinical care remain disappointingly rare.…”