Effect of Integrating Machine Learning Mortality Estimates With Behavioral Nudges to Clinicians on Serious Illness Conversations Among Patients With Cancer

Abstract

IMPORTANCE Serious illness conversations (SICs) are structured conversations between clinicians and patients about prognosis, treatment goals, and end-of-life preferences. Interventions that increase the rate of SICs between oncology clinicians and patients may improve goal-concordant care and patient outcomes. OBJECTIVE To determine the effect of a clinician-directed intervention integrating machine learning mortality predictions with behavioral nudges on motivating clinician-patient SICs. DESIGN, SETTING, AND PARTICIPANTS This stepped-wedge cluster randomized clinical trial was conducted across 20 weeks (from June 17 to November 1, 2019) at 9 medical oncology clinics (8 subspecialty oncology and 1 general oncology clinics) within a large academic health system in Pennsylvania. Clinicians at the 2 smallest subspecialty clinics were grouped together, resulting in 8 clinic groups randomly assigned to the 4 intervention wedge periods. Included participants in the intention-to-treat analyses were 78 oncology clinicians who received SIC training and their patients (N = 14 607) who had an outpatient oncology encounter during the study period. INTERVENTIONS (1) Weekly emails to oncology clinicians with SIC performance feedback and peer comparisons; (2) a list of up to 6 high-risk patients (Ն10% predicted risk of 180-day mortality) scheduled for the next week, estimated using a validated machine learning algorithm; and (3) opt-out text message prompts to clinicians on the patient's appointment day to consider an SIC. Clinicians in the control group received usual care consisting of weekly emails with cumulative SIC performance. MAIN OUTCOMES AND MEASURES Percentage of patient encounters with an SIC in the intervention group vs the usual care (control) group. RESULTS The sample consisted of 78 clinicians and 14 607 patients. The mean (SD) age of patients was 61.9 (14.2) years, 53.7% were female, and 70.4% were White. For all encounters, SICs were conducted among 1.3% in the control group and 4.6% in the intervention group, a significant difference (adjusted difference in percentage points, 3.3; 95% CI, 2.3-4.5; P < .001). Among 4124 high-risk patient encounters, SICs were conducted among 3.6% in the control group and 15.2% in the intervention group, a significant difference (adjusted difference in percentage points, 11.6; 95% CI, 8.2-12.5; P < .001). CONCLUSIONS AND RELEVANCE In this stepped-wedge cluster randomized clinical trial, an intervention that delivered machine learning mortality predictions with behavioral nudges to oncology clinicians significantly increased the rate of SICs among all patients and among patients with high mortality risk who were targeted by the intervention. Behavioral nudges combined with machine learning mortality predictions can positively influence clinician behavior and may be applied more broadly to improve care near the end of life.

Clinical Utility of Automatable Prediction Models for Improving Palliative and End-Of-Life Care Outcomes: Towards Routine Decision Analysis Before Implementation

“…Future studies can refine on decision analysis based on this retroactive feedback, like including model-independent effects from behavioural economic–inspired co-interventions. [47]…”

Implementing automated prognostic models to inform palliative care: more than just the algorithm

“…24 For example, using an 'opt-out' approach for palliative care referral may make the optimal choice the path of least resistance, increasing uptake among clinicians. 16 These approaches will need to be balanced against rising clinician alert fatigue 25 and resource constraints.…”

Trajectories of mortality risk among patients with cancer and associated end-of-life utilization

“…Prognostic inaccuracy among clinicians contributes to low-quality EOL care, including delayed hospice utilization and increased acute care utilization close to death, for patients with serious illnesses, such as cancer 3 , 4 . Machine learning (ML) algorithms outperform traditional tools used for prognostication and may facilitate earlier clinician–patient discussions about hospice enrollment, discontinuation of therapy, or other management decisions 5 . Most research to date has reported on the performance of static predictions of mortality risk from ML algorithms 5 – 7 .…”

“…Machine learning (ML) algorithms outperform traditional tools used for prognostication and may facilitate earlier clinician–patient discussions about hospice enrollment, discontinuation of therapy, or other management decisions 5 . Most research to date has reported on the performance of static predictions of mortality risk from ML algorithms 5 – 7 . However, patients’ risk of mortality may change over time in non-linear patterns.…”

“…To calculate mortality risk, we used a previously trained gradient boosting machine (GBM), an ensemble ML algorithm, based on 559 structured electronic health record (EHR) variables for 26,525 patients with cancer who were seen at 11 academic or community medical oncology practices within a large academic cancer center in 2016 8 . The training and prospective validation of this algorithm among a more recent cohort seen at 18 oncology practices have been previously published 5 , 8 . This algorithm was integrated into our EHR in 2018.…”