Development of a Machine Learning Model Using Limited Features to Predict 6-Month Mortality at Treatment Decision Points for Patients With Advanced Solid Tumors

Abstract

PURPOSE Patients with advanced solid tumors may receive intensive treatments near the end of life. This study aimed to create a machine learning (ML) model using limited features to predict 6-month mortality at treatment decision points (TDPs). METHODS We identified a cohort of adults with advanced solid tumors receiving care at a major cancer center from 2014 to 2020. We identified TDPs for new lines of therapy (LoTs) and confirmed mortality at 6 months after a TDP. Using extreme gradient boosting, ML models were developed, which used or derived features from a limited set of electronic health record data considering the literature, clinical relevance, variability, availability, and predictive importance using Shapley additive explanations scores. We predicted and observed 6-month mortality after a TDP and assessed a risk stratification strategy with different risk thresholds to support communication of chance of survival. RESULTS Four thousand one hundred ninety-two patients were included. Patients had 7,056 TDPs, for which the 6-month mortality increased from 17.9% to 46.7% after starting first to sixth LoT, respectively. On the basis of internal validation, models using both 111 (Full) or 45 (Limited-45) features accurately predicted 6-month mortality (area under the curve ≥ 0.80). Using a 0.3 risk threshold in the Limited-45 model, the observed 6-month survival was 34% (95% CI, 28 to 40) versus 81% (95% CI, 81 to 82) among those classified with low or higher chance of survival, respectively. The positive predictive value of the Limited-45 model was 0.66 (95% CI, 0.60 to 0.72). CONCLUSION We developed and validated a ML model using a limited set of 45 features readily derived from electronic health record data to predict 6-month prognosis in patients with advanced solid tumors. The model output may support shared decision making as patients consider the next LoT.

External Validation of a Machine Learning Model to Predict 6-Month Mortality for Patients With Advanced Solid Tumors

“…Retrospective cohort identification, TDP extraction, and ML model details are summarized in the eMethods in Supplement 1 and replicated processes during model development. 2 The ML model was originally trained on TDPs between June 1, 2014, and June 1, 2020. It was evaluated on newly identified TDPs between June 2, 2020, and April 12, 2022, including subset analyses (eMethods in Supplement 1).…”

“…We assessed model performance using area under the curve (AUC) and determined positive predictive value, negative predictive value, sensitivity, and specificity at the predetermined risk threshold of 0.3 so that approximately 1 in 3 patients classified as low chance of surviving were alive after 6 months, consistent with perceptions of clinical experts. 2 Quality metrics, such as referrals for palliative care or hospice, hospitalization rates, and mean length of stay, were calculated for patients classified with low chance of survival. Analyses were performed using Python version 3.9.12, scikit-learn 1.0.2, and scipy 1.7.3 (Python Software Foundation) with 1-sided P < .05 to determine statistical significance.…”

“…In this prognostic study, external validation was performed using EHR data extracted from the University of Utah Health enterprise data warehouse on October 12, 2022. Retrospective cohort identification, TDP extraction, and ML model details are summarized in the eMethods in Supplement 1 and replicated processes during model development . The ML model was originally trained on TDPs between June 1, 2014, and June 1, 2020.…”

“…Population characteristics between development and validation data were compared. We assessed model performance using area under the curve (AUC) and determined positive predictive value, negative predictive value, sensitivity, and specificity at the predetermined risk threshold of 0.3 so that approximately 1 in 3 patients classified as low chance of surviving were alive after 6 months, consistent with perceptions of clinical experts . Quality metrics, such as referrals for palliative care or hospice, hospitalization rates, and mean length of stay, were calculated for patients classified with low chance of survival.…”

“…1 To facilitate effective conversations at treatment decision points (TDPs) between practitioners and patients with advanced cancer, team members previously developed and internally validated an ML model that classifies patients into low or likely chance of surviving the next 6 months if initiating a new line of therapy (LOT). 2 The unique model uses 45 features derived from electronic health record (EHR) data that are implementable using the Fast Healthcare Interoperability Resources standard, 2 supporting interpretability and interoperability across health systems. The model is expected to be implemented within a tool to transparently communicate and explain ML-based prognosis.…”