Supporting Acute Advance Care Planning with Precise, Timely Mortality Risk Predictions

Wang, Erwin; Major, Vincent J.; Adler, Nicole; Hauck, Kevin; Austrian, Jonathan; Aphinyanaphongs, Yindalon; Horwitz, Leora I.

doi:10.1056/cat.20.0655

Cited by 14 publications

(24 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, this analysis suggests mechanisms by which ML-based interventions may increase advance care planning in previous trials [6,13,14]. One possible reason for variable response to an ML-based intervention observed in this study is variation in cognitive workload.…”

Section: Discussionmentioning

confidence: 75%

“…One pragmatic randomized control trial found that an ML-based prompt to oncology clinicians increased rates of ACPs from 3% to 15% of all patients at a large academic cancer center [5,6]. Similar ML-based interventions have been shown to increase ACP documentation [13], reduce length of stay, and increase home palliative care referrals [14]. However, clinicians have heterogeneous responses to such strategies [11], and the efficacy of such interventions across oncology clinician subgroups is not well understood.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Oncologist phenotypes and associations with response to a machine learning-based intervention to increase advance care planning: Secondary analysis of a randomized clinical trial

Manz

Liu

et al. 2022

PLoS ONE

View full text Add to dashboard Cite

Background While health systems have implemented multifaceted interventions to improve physician and patient communication in serious illnesses such as cancer, clinicians vary in their response to these initiatives. In this secondary analysis of a randomized trial, we identified phenotypes of oncology clinicians based on practice pattern and demographic data, then evaluated associations between such phenotypes and response to a machine learning (ML)-based intervention to prompt earlier advance care planning (ACP) for patients with cancer. Methods and findings Between June and November 2019, we conducted a pragmatic randomized controlled trial testing the impact of text message prompts to 78 oncology clinicians at 9 oncology practices to perform ACP conversations among patients with cancer at high risk of 180-day mortality, identified using a ML prognostic algorithm. All practices began in the pre-intervention group, which received weekly emails about ACP performance only; practices were sequentially randomized to receive the intervention at 4-week intervals in a stepped-wedge design. We used latent profile analysis (LPA) to identify oncologist phenotypes based on 11 baseline demographic and practice pattern variables identified using EHR and internal administrative sources. Difference-in-differences analyses assessed associations between oncologist phenotype and the outcome of change in ACP conversation rate, before and during the intervention period. Primary analyses were adjusted for patients’ sex, age, race, insurance status, marital status, and Charlson comorbidity index. The sample consisted of 2695 patients with a mean age of 64.9 years, of whom 72% were White, 20% were Black, and 52% were male. 78 oncology clinicians (42 oncologists, 36 advanced practice providers) were included. Three oncologist phenotypes were identified: Class 1 (n = 9) composed primarily of high-volume generalist oncologists, Class 2 (n = 5) comprised primarily of low-volume specialist oncologists; and 3) Class 3 (n = 28), composed primarily of high-volume specialist oncologists. Compared with class 1 and class 3, class 2 had lower mean clinic days per week (1.6 vs 2.5 [class 3] vs 4.4 [class 1]) a higher percentage of new patients per week (35% vs 21% vs 18%), higher baseline ACP rates (3.9% vs 1.6% vs 0.8%), and lower baseline rates of chemotherapy within 14 days of death (1.4% vs 6.5% vs 7.1%). Overall, ACP rates were 3.6% in the pre-intervention wedges and 15.2% in intervention wedges (11.6 percentage-point difference). Compared to class 3, oncologists in class 1 (adjusted percentage-point difference-in-differences 3.6, 95% CI 1.0 to 6.1, p = 0.006) and class 2 (adjusted percentage-point difference-in-differences 12.3, 95% confidence interval [CI] 4.3 to 20.3, p = 0.003) had greater response to the intervention. Conclusions Patient volume and time availability may be associated with oncologists’ response to interventions to increase ACP. Future interventions to prompt ACP should prioritize making time available for such conversations between oncologists and their patients.

show abstract

Section: Discussionmentioning

confidence: 75%

Section: Introductionmentioning

confidence: 99%

Oncologist phenotypes and associations with response to a machine learning-based intervention to increase advance care planning: Secondary analysis of a randomized clinical trial

Manz

Liu

et al. 2022

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…[13] NYU Langone's model performance, with an AUC-PR of 28%, was enough to achieve good rates of physician agreement with the alerts and greater use of ACPs. [14] Therefore, we hoped to achieve a similar level of performance in with our model in our mixed-rurality population and maintain that performance over time despite changing conditions. COVID-19 created signi cant systemic change in the healthcare, and systemic change often leads to performance degradation in machine learned models.…”

Section: Discussionmentioning

confidence: 92%

“…This work was inspired by the studies out of NYU Langone demonstrating the performance and impact of their 60-day mortality prediction model which was intended to encourage ACP discussions [14] as well as to encourage appropriate patient referrals to supportive and palliative care. [13] NYU Langone's model performance, with an AUC-PR of 28%, was enough to achieve good rates of physician agreement with the alerts and greater use of ACPs.…”

Section: Discussionmentioning

confidence: 99%

“…[7] Predicting mortality using clinician gestalt alone may have limited accuracy, but combining gestalt with a predictive model may be synergistic. [14] Therefore, to help prioritize ACPs when resources are limited, and to encourage clinicians to have ACPs in those more likely to bene t, we developed a model to predict mortality occurring 5-to-90 days after the start of an inpatient admission. The 5-90 day window was chosen to be not too short and not too long ("Goldilocks"), providing at least 4 days to have the discussion (average length of an inpatient stay [15]), while also creating enough urgency to stimulate the ACP.…”

Section: Background and Signi Cancementioning

confidence: 99%

See 1 more Smart Citation

Can a “goldilocks” Mortality Predictor Perform Consistently Across Time and Equitably Across Populations?

Handler

Lee

Iyer

et al. 2022

Preprint

View full text Add to dashboard Cite

Objective: Advance care planning (ACP) facilitates end-of-life care, yet many die without one. Timely and accurate mortality prediction may encourage ACP. Therefore, we assessed performance equity and consistency for a novel 5-to-90-day mortality predictor. Methods: Predictions were made for the first day of included adult inpatient admissions on a retrospective dataset. Performance was assessed across various demographies, geographies, and timeframes. Results: AUC-PR remained at 29% both pre- and during COVID. Pre-COVID-19 recall and precision were 58% and 25% respectively at the 12.5% cutoff, and 12% and 44% at the 37.5% cutoff. During COVID-19, recall and precision were 59% and 26% at the 12.5% cutoff, and 11% and 43% at the 37.5% cutoff. Pre-COVID, recall dropped at both cutoffs if recent data was not made available to the model; and compared to the overall population, recall was lower at the 12.5% cutoff in the White, non-Hispanic subgroup and at both cutoffs in the rural subgroup. During COVID-19, precision at the 12.5% cutoff was lower than that of the overall population for the non-White and non-White female subgroups. No other statistically significant differences were seen between subgroups and the corresponding overall population. Conclusions: Overall predictive performance during the pandemic was unchanged from pre-pandemic performance. Although some comparisons (especially precision at the 37.5% cutoff) were underpowered, precision at the 12.5% cutoff was equitable across most demographies, regardless of the pandemic. Mortality prediction to prioritize ACP conversations can be provided consistently and equitably across many studied timeframes, geographies, and demographies.

show abstract

Advanced Care Planning for Hospitalized Patients Following Clinician Notification of Patient Mortality by a Machine Learning Algorithm

et al. 2023

View full text Add to dashboard Cite

ImportanceGoal-concordant care is an ongoing challenge in hospital settings. Identification of high mortality risk within 30 days may call attention to the need to have serious illness conversations, including the documentation of patient goals of care.ObjectiveTo examine goals of care discussions (GOCDs) in a community hospital setting with patients identified as having a high risk of mortality by a machine learning mortality prediction algorithm.Design, Setting, and ParticipantsThis cohort study took place at community hospitals within 1 health care system. Participants included adult patients with a high risk of 30-day mortality who were admitted to 1 of 4 hospitals between January 2 and July 15, 2021. Patient encounters of inpatients in the intervention hospital where physicians were notified of the computed high risk mortality score were compared with patient encounters of inpatients in 3 community hospitals without the intervention (ie, matched control).InterventionPhysicians of patients with a high risk of mortality within 30 days received notification and were encouraged to arrange for GOCDs.Main Outcomes and MeasuresThe primary outcome was the percentage change of documented GOCDs prior to discharge. Propensity-score matching was completed on a preintervention and postintervention period using age, sex, race, COVID-19 status, and machine learning-predicted mortality risk scores. A difference-in-difference analysis validated the results.ResultsOverall, 537 patients were included in this study with 201 in the preintervention period (94 in the intervention group; 104 in the control group) and 336 patients in the postintervention period. The intervention and control groups included 168 patients per group and were well-balanced in age (mean [SD], 79.3 [9.60] vs 79.6 [9.21] years; standardized mean difference [SMD], 0.03), sex (female, 85 [51%] vs 85 [51%]; SMD, 0), race (White patients, 145 [86%] vs 144 [86%]; SMD 0.006), and Charlson comorbidities (median [range], 8.00 [2.00-15.0] vs 9.00 [2.00 to 19.0]; SMD, 0.34). Patients in the intervention group from preintervention to postintervention period were associated with being 5 times more likely to have documented GOCDs (OR, 5.11 [95% CI, 1.93 to 13.42]; P = .001) by discharge compared with matched controls, and GOCD occurred significantly earlier in the hospitalization in the intervention patients as compared with matched controls (median, 4 [95% CI, 3 to 6] days vs 16 [95% CI, 15 to not applicable] days; P &lt; .001). Similar findings were observed for Black patient and White patient subgroups.Conclusions and RelevanceIn this cohort study, patients whose physicians had knowledge of high-risk predictions from machine learning mortality algorithms were associated with being 5 times more likely to have documented GOCDs than matched controls. Additional external validation is needed to determine if similar interventions would be helpful at other institutions.

show abstract

Supporting Acute Advance Care Planning with Precise, Timely Mortality Risk Predictions

Cited by 14 publications

References 31 publications

Oncologist phenotypes and associations with response to a machine learning-based intervention to increase advance care planning: Secondary analysis of a randomized clinical trial

Oncologist phenotypes and associations with response to a machine learning-based intervention to increase advance care planning: Secondary analysis of a randomized clinical trial

Can a “goldilocks” Mortality Predictor Perform Consistently Across Time and Equitably Across Populations?

Advanced Care Planning for Hospitalized Patients Following Clinician Notification of Patient Mortality by a Machine Learning Algorithm

Contact Info

Product

Resources

About