Artificial intelligence sepsis prediction algorithm learns to say “I don’t know”

Abstract

Sepsis is a leading cause of morbidity and mortality worldwide. Early identification of sepsis is important as it allows timely administration of potentially life-saving resuscitation and antimicrobial therapy. We present COMPOSER (COnformal Multidimensional Prediction Of SEpsis Risk), a deep learning model for the early prediction of sepsis, specifically designed to reduce false alarms by detecting unfamiliar patients/situations arising from erroneous data, missingness, distributional shift and data drifts. COMPOSER flags these unfamiliar cases as indeterminate rather than making spurious predictions. Six patient cohorts (515,720 patients) curated from two healthcare systems in the United States across intensive care units (ICU) and emergency departments (ED) were used to train and externally and temporally validate this model. In a sequential prediction setting, COMPOSER achieved a consistently high area under the curve (AUC) (ICU: 0.925–0.953; ED: 0.938–0.945). Out of over 6 million prediction windows roughly 20% and 8% were identified as indeterminate amongst non-septic and septic patients, respectively. COMPOSER provided early warning within a clinically actionable timeframe (ICU: 12.2 [3.2 22.8] and ED: 2.1 [0.8 4.5] hours prior to first antibiotics order) across all six cohorts, thus allowing for identification and prioritization of patients at high risk for sepsis.

Multicenter Validation of a Machine Learning Algorithm for Diagnosing Pediatric Patients with Multisystem Inflammatory Syndrome and Kawasaki Disease

“…We selected the model with the highest accuracy during the 10-fold cross-validation at each stage to use in the final model. To ensure model generalizability and similar performance across external sites, we constructed a conformal prediction framework using the training samples from the final model 12 . Briefly, we selected the parsimonious combination of missingness and risk score with the highest weighted F1 score to construct the conformal trust sets ( Supplementary Figure 2, Supplementary Table 1 ).…”

“…In KIDMATCH, we implemented a conformal prediction framework to reject out of distribution samples 12 and separated the classification into two stages ( Figure 1 ). If a test sample was rejected by the conformal prediction framework, no prediction was calculated.…”

“…The model in Stage 2 was trained to further classify patients falling into the “other” category as KD or other pediatric febrile illness. Since KD and MIS-C data distributions may vary across different sites, we incorporated a conformal prediction framework within the model 12 . Conformal prediction reduces false alarms by identifying unfamiliar samples in new patient populations when compared to the training cohort and assigns indeterminate labels rather than making spurious predictions.…”

Preparing for a New World: Making Friends with Digital Health

“…1A ). 12 13 This is because a model forecasts an individual’s risk of a given outcome on the basis of a group of individuals with similar features. Therefore, we need to consider differences between the environment where a digital health solution was developed and that in which it intends to be implemented to determine whether it can be adopted successfully.…”

“…The above-described issues have prompted efforts to reduce errors by calculating uncertainty. 12 In the future, when numerous digital solutions will be part of standard health care, it may be difficult for users to understand the specific features of individual solutions. Therefore, models that can judge whether a given clinical environment is compatible with a digital health solution will be helpful.…”