Evaluating equity in performance of an electronic health record-based 6-month mortality risk model to trigger palliative care consultation: a retrospective model validation analysis

Abstract

ObjectiveEvaluate predictive performance of an electronic health record (EHR)-based, inpatient 6-month mortality risk model developed to trigger palliative care consultation among patient groups stratified by age, race, ethnicity, insurance and socioeconomic status (SES), which may vary due to social forces (eg, racism) that shape health, healthcare and health data.DesignRetrospective evaluation of prediction model.SettingThree urban hospitals within a single health system.ParticipantsAll patients ≥18 years admitted between 1 January and 31 December 2017, excluding observation, obstetric, rehabilitation and hospice (n=58 464 encounters, 41 327 patients).Main outcome measuresGeneral performance metrics (c-statistic, integrated calibration index (ICI), Brier Score) and additional measures relevant to health equity (accuracy, false positive rate (FPR), false negative rate (FNR)).ResultsFor black versus non-Hispanic white patients, the model’s accuracy was higher (0.051, 95% CI 0.044 to 0.059), FPR lower (−0.060, 95% CI −0.067 to −0.052) and FNR higher (0.049, 95% CI 0.023 to 0.078). A similar pattern was observed among patients who were Hispanic, younger, with Medicaid/missing insurance, or living in low SES zip codes. No consistent differences emerged in c-statistic, ICI or Brier Score. Younger age had the second-largest effect size in the mortality prediction model, and there were large standardised group differences in age (eg, 0.32 for non-Hispanic white versus black patients), suggesting age may contribute to systematic differences in the predicted probabilities between groups.ConclusionsAn EHR-based mortality risk model was less likely to identify some marginalised patients as potentially benefiting from palliative care, with younger age pinpointed as a possible mechanism. Evaluating predictive performance is a critical preliminary step in addressing algorithmic inequities in healthcare, which must also include evaluating clinical impact, and governance and regulatory structures for oversight, monitoring and accountability.

Clinical decision-making and algorithmic inequality

“…The 'insensitivity to impact' of traditional performance measures 6 means that the balance of harm and benefit of two hypothetical predictive models with identical overall performance scores may be very different for a clinical application that results in significant unnecessary intervention (eg, surgery) versus one that results in harmful undertreatment. As Teeple and colleagues 8 nicely prove here, it is not sufficient to determine that an algorithm is equivalently accurate in a disadvantaged subgroup to rule out inequality resulting from its use.…”

“…6 Decision support algorithms based on such models may then propagate the kind of 'race-based' decisions predicated on historical inequality described by Vyas et al The mortality prediction among 41 327 patients admitted in 2017 was evaluated in different marginalised subpopulations, including race, Hispanic ethnicity, health insurance status, household income and education level, none of which were predictors in 'Palliative Connect'. Teeple and colleagues 8 found that false negatives (ie, mortality risk underestimated) were more common in patients with younger age, lower income, black race or Hispanic ethnicity and lower education (eg, false negative rate was 4.9% higher for black vs non-Hispanic white patients). Conversely, false positives (ie, mortality risk overestimated) were more common in older, white/Asian and male patients with insurance and hence these patients were more likely to be considered for palliative care referral (eg, false positive rate lower by 6.0% in blacks vs non-Hispanic whites).…”

“…Issues with the explicit use of race in models lead Vyas et al 7 to recommend that three conditions are met before it is included in algorithms: first, that there is no evidence of confounding between race and more specific factors; second, that there is a plausible causative mechanism; and third, that including race will relieve rather than exacerbate health inequalities. Teeple and colleagues 8 demonstrate that these considerations are important even if race (or indicators of other disadvantaged groups) is not directly represented in clinical algorithms as other factors included in the model may still be correlated with race. Advancing health equity involves identifying inadequate care due to socioeconomic factors as a patient safety issue and developing performance measures that highlight inequality.…”

“…Advancing health equity involves identifying inadequate care due to socioeconomic factors as a patient safety issue and developing performance measures that highlight inequality. 2 14 Teeple and colleagues 8 have identified how relevant these goals are in the context of clinical decision algorithms.…”