Validation of a Proprietary Deterioration Index Model and Performance in Hospitalized Adults

Abstract

ImportanceThe Deterioration Index (DTI), used by hospitals for predicting patient deterioration, has not been extensively validated externally, raising concerns about performance and equitable predictions.ObjectiveTo locally validate DTI performance and assess its potential for bias in predicting patient clinical deterioration.Design, Setting, and ParticipantsThis retrospective prognostic study included 13 737 patients admitted to 8 heterogenous Midwestern US hospitals varying in size and type, including academic, community, urban, and rural hospitals. Patients were 18 years or older and admitted between January 1 and May 31, 2021.ExposureDTI predictions made every 15 minutes.Main Outcomes and MeasuresDeterioration, defined as the occurrence of any of the following while hospitalized: mechanical ventilation, intensive care unit transfer, or death. Performance of the DTI was evaluated using area under the receiver operating characteristic curve (AUROC) and area under the precision recall curve (AUPRC). Bias measures were calculated across demographic subgroups.ResultsA total of 5 143 513 DTI predictions were made for 13 737 patients across 14 834 hospitalizations. Among 13 918 encounters, the mean (SD) age of patients was 60.3 (19.2) years; 7636 (54.9%) were female, 11 345 (81.5%) were White, and 12 392 (89.0%) were of other ethnicity than Hispanic or Latino. The prevalence of deterioration was 10.3% (n = 1436). The DTI produced AUROCs of 0.759 (95% CI, 0.756-0.762) at the observation level and 0.685 (95% CI, 0.671-0.700) at the encounter level. Corresponding AUPRCs were 0.039 (95% CI, 0.037-0.040) at the observation level and 0.248 (95% CI, 0.227-0.273) at the encounter level. Bias measures varied across demographic subgroups and were 14.0% worse for patients identifying as American Indian or Alaska Native and 19.0% worse for those who chose not to disclose their ethnicity.Conclusions and RelevanceIn this prognostic study, the DTI had modest ability to predict patient deterioration, with varying degrees of performance at the observation and encounter levels and across different demographic groups. Disparate performance across subgroups suggests the need for more transparency in model training data and reinforces the need to locally validate externally developed prediction models.

Enhanced family history-based algorithms increase the identification of individuals meeting criteria for genetic testing of hereditary cancer syndromes but would not reduce disparities on their own

“…[ 18 ] A study examining demographic disparities in algorithmic performance for detecting in-hospital patient deterioration has found analogous discrepancies post-hoc, years after the algorithm’s widespread implementation, and by researchers not initially involved in its development. [ 19 ] This kind of validation is rarely done as a part of algorithm development, before algorithms are widely implemented in clinical settings. The present study illustrates a method that could be used as a part of algorithm development to systematically quantify and compare differences by demographic characteristics resulting from multiple EHR data-driven algorithms.…”