Predicting Intensive Care Delirium with Machine Learning: Model Development and External Validation

Abstract

Background Delirium poses significant risks to patients, but countermeasures can be taken to mitigate negative outcomes. Accurately forecasting delirium in ICU patients could guide proactive intervention. Our primary objective was to predict ICU delirium by applying machine learning to clinical and physiological data routinely collected in electronic health records. Methods Two prediction models were trained and tested using a multi-center database (years of data collection 2014-15), and externally validated on two single-center databases (2001-2012 and 2008-2019). The primary outcome variable was delirium defined as a positive Confusion Assessment Method for the ICU screen, or an Intensive Care Delirium Screening Checklist ≥4. The first model, named “24-hour model”, used data from the 24 hours following ICU admission to predict delirium any time afterwards. The second model designated “dynamic model", predicted the onset of delirium up to 12 hours in advance. Model performance was compared to results using features from a widely-cited reference model. Results For the 24-hour model, delirium was identified in 2,536/18,305 (13.9%), 768/5,299 (14.5%), and 5,955/36,194 (11.9%) of patient stays respectively in the development sample and two validation samples. For the 12-hour lead time dynamic model, delirium was identified in 3,791/22,234 (17.0%), 994/6,166 (16.1%), and 5,955/28,440 (20.9%) patient stays, respectively. Mean AUC (95% CI) for the first 24-hour model was 0.785 (0.769, 0.801), significantly higher than the modified reference model with AUC of 0.730 (0.704, 0.757). The dynamic model had a mean AUC of 0.845 (0.831, 0.859) when predicting delirium 12 hours in advance. Calibration was similar in both models (mean Brier Score [95% CI] 0.102 [0.097, 0.108] and 0.111 [0.106, 0.116]). Model discrimination and calibration were maintained when tested on the validation datasets. Conclusions Machine learning models trained with clinical and physiological data predict ICU delirium and support dynamic time-sensitive forecasting.

Frailty in the prediction of delirium in the intensive care unit: A secondary analysis of the Deli study

“…Such an approach would need to be tested in a randomized trial of absolute risk communication and could explore innovative approaches based on dynamic machine learning models. 44 In conclusion, we have been able to show that both the cumulative deficits frailty index and clinical frailty scale predict an acute episode of delirium among adults admitted to intensive care.…”

“…Frailty status has been identified as an important predictor in the surgical setting 41,42 and among trauma patients, 43 however, not in the general intensive care population. An important innovation in the intensive care setting appears to be the use of machine learning and the use of dynamic prediction models 44 . These approaches appear to be able to create dynamic risk prediction estimates, that change during an individual's intensive care stay.…”

“…An important innovation in the intensive care setting appears to be the use of machine learning and the use of dynamic prediction models. 44 These approaches appear to be able to create dynamic risk prediction estimates, that change during an individual's intensive care stay.…”

Evaluating the performance of machine learning methods for risk estimation of delirium in patients hospitalized from the emergency department

“…[10][11][12] There is a need to identify an optimal screening strategy for delirium beyond cognitive assessment because, until we have it, delirium will likely remain an elusive diagnosis. In recent years, numerous studies have developed models in the hospital setting for estimating the risk of delirium in postoperative patients [13][14][15][16][17][18][19][20][21][22] and ICU patients, [23][24][25][26][27][28] but limited work has focused on the ED patient population. 3,29,30 Our objective was to leverage electronic health records to identify a clinically valuable risk estimation model for positive delirium screens in patients admitted from the ED to inpatient units.…”

“…There is a need to identify an optimal screening strategy for delirium beyond cognitive assessment because, until we have it, delirium will likely remain an elusive diagnosis. In recent years, numerous studies have developed models in the hospital setting for estimating the risk of delirium in postoperative patients 13–22 and ICU patients, 23–28 but limited work has focused on the ED patient population 3,29,30 …”