Expected clinical utility of automatable prediction models for improving palliative and end-of-life care outcomes: Toward routine decision analysis before implementation

Abstract

Objective The study sought to evaluate the expected clinical utility of automatable prediction models for increasing goals-of-care discussions (GOCDs) among hospitalized patients at the end of life (EOL). Materials and Methods We built a decision model from the perspective of clinicians who aim to increase GOCDs at the EOL using an automated alert system. The alternative strategies were 4 prediction models—3 random forest models and the Modified Hospital One-year Mortality Risk model—to generate alerts for patients at a high risk of 1-year mortality. They were trained on admissions from 2011 to 2016 (70 788 patients) and tested with admissions from 2017-2018 (16 490 patients). GOCDs occurring in usual care were measured with code status orders. We calculated the expected risk difference (beneficial outcomes with alerts minus beneficial outcomes without alerts among those at the EOL), the number needed to benefit (number of alerts needed to increase benefit over usual care by 1 outcome), and the net benefit (benefit minus cost) of each strategy. Results Models had a C-statistic between 0.79 and 0.86. A code status order occurred during 2599 of 3773 (69%) hospitalizations at the EOL. At a risk threshold corresponding to an alert prevalence of 10%, the expected risk difference ranged from 5.4% to 10.7% and the number needed to benefit ranged from 5.4 to 10.9 alerts. Using revealed preferences, only 2 models improved net benefit over usual care. A random forest model with diagnostic predictors had the highest expected value, including in sensitivity analyses. Discussion Prediction models with acceptable predictive validity differed meaningfully in their ability to improve over usual decision making. Conclusions An evaluation of clinical utility, such as by using decision curve analysis, is recommended after validating a prediction model because metrics of model predictiveness, such as the C-statistic, are not informative of clinical value.

Machine Learning to Allocate Palliative Care Consultations During Cancer Treatment

“…When trained to predict shorter-term mortality, our system achieved an AUROC of 0.826-0.884 (Data Supplement, Table S6), similar to previous studies, which reported an AUROC of 0.868 for 30-day mortality 17 and 0.81 for 90-day mortality. 14 For predicting longer-term mortality, previous studies reported a wide range of AUROCs, [29][30][31][32] ranging from 0.78 32 to 0.88. 29 For example, the system of Parikh et al 29 had an AUROC of 0.88, but the prevalence of 180-day mortality was 4%, reflecting the inclusion of early-stage curable patients who are unlikely to die and would not be considered for PC.…”

Recent advancement in integrating artificial intelligence and information technology with real‐world data for clinical decision‐making in China: A scoping review

“…To make AI‐based CDSSs clinically useful, model evaluation considering clinical utility should be embedded into the model building process. Decision curve analysis that quantifies the net benefit to inform clinical value has been well‐established but has yet to be exploited in building predictive models, 82,106,107 especially in China. Net benefit incorporated the clinical outcomes of the decisions made based on the built model, providing more clinical insights into whether the benefits would outweigh the risks 106 .…”

“…105 To make AI-based CDSSs clinically useful, model evaluation considering clinical utility should be embedded into the model building process. Decision curve analysis that quantifies the net benefit to inform clinical value has been well-established but has yet to be exploited in building predictive models, 82,106,107 especially in China.…”

Advanced Care Planning for Hospitalized Patients Following Clinician Notification of Patient Mortality by a Machine Learning Algorithm

“…The potential of machine learning predictions to expand SIC has been trialed at a handful of academic hospitals in recent years, with studies showing challenging feasibility but likely improved clinical outcomes. However, firm conclusions have been limited by the absence of controls and the recent variability in health care and patient mix caused by the COVID-19 pandemic . We have recently published the structure of an accurate machine learning algorithm for predicting short-term mortality in a modern inpatient population .…”

“…However, firm conclusions have been limited by the absence of controls and the recent variability in health care and patient mix caused by the COVID-19 pandemic. 25 , 26 , 27 , 28 , 29 , 30 We have recently published the structure of an accurate machine learning algorithm for predicting short-term mortality in a modern inpatient population. 31 By implementing this algorithm in a community hospital with propensity-matched controls, we sought to investigate whether a mortality risk-targeted EHR prompt was associated with increased inpatient goals of care documentation.…”