Machine learning versus physicians’ prediction of acute kidney injury in critically ill adults: a prospective evaluation of the AKIpredictor

Abstract

Background Early diagnosis of acute kidney injury (AKI) is a major challenge in the intensive care unit (ICU). The AKIpredictor is a set of machine-learning-based prediction models for AKI using routinely collected patient information, and accessible online. In order to evaluate its clinical value, the AKIpredictor was compared to physicians’ predictions. Methods Prospective observational study in five ICUs of a tertiary academic center. Critically ill adults without end-stage renal disease or AKI upon admission were considered for enrollment. Using structured questionnaires, physicians were asked upon admission, on the first morning, and after 24 h to predict the development of AKI stages 2 or 3 (AKI-23) during the first week of ICU stay. Discrimination, calibration, and net benefit of physicians’ predictions were compared against the ones by the AKIpredictor. Results Two hundred fifty-two patients were included, 30 (12%) developed AKI-23. In the cohort of patients with predictions by physicians and AKIpredictor, the performance of physicians and AKIpredictor were respectively upon ICU admission, area under the receiver operating characteristic curve (AUROC) 0.80 [0.69–0.92] versus 0.75 [0.62–0.88] ( n = 120, P = 0.25) with net benefit in ranges 0–26% versus 0–74%; on the first morning, AUROC 0.94 [0.89–0.98] versus 0.89 [0.82–0.97] ( n = 187, P = 0.27) with main net benefit in ranges 0–10% versus 0–48%; after 24 h, AUROC 0.95 [0.89–1.00] versus 0.89 [0.79–0.99] ( n = 89, P = 0.09) with main net benefit in ranges 0–67% versus 0–50%. Conclusions The machine-learning-based AKIpredictor achieved similar discriminative performance as physicians for prediction of AKI-23, and higher net benefit overall, because physicians overestimated the risk of AKI. This suggests an added value of the systematic risk stratification by the AKIpredictor to physicians’ predictions, in particular to select high-risk patients or reduce false positives in studies evaluating new and potentially harmful therapies. Due to the low event rate, future studies are needed to validate these findings. Trial registration ClinicalTrials.gov, NCT03574896 registration date: July 2nd, 2018 Electronic supplementary material The online version of this article (10.1186/s13054-019-2563-x) contains supplementary material, which is available to authorized users.

Early Recognition of Burn- and Trauma-Related Acute Kidney Injury: A Pilot Comparison of Machine Learning Techniques

“…This study evaluates the generalizability of a burn population derived ML algorithm for predicting AKI in a mixed burn and non-burned trauma population. Overall, ML is clearly able to provide unique advantages in the context of AKI including the potential to be highly automated via electronic medical record systems, and as observed in previous and current studies, enable early classification of subtle changes for predicting AKI [24][25][26][27] . Kate et al used LR, SVM, decision trees, and naïve Bayes to detect undiagnosed AKI in a large population of hospitalized elderly (age >60 years) patients 25 .…”

“…The study reported area under the ROC curves ranging from 0.66 to 0.74. More recent studies compared the performance of ML versus physician prediction of AKI based on KDIGO criteria to achieve area under the ROC curves of 0.75 and 0.80 respectively for data presented at ICU admission 27 . Optimal performance was achieved with data after 24 hours with area under the ROC curve of 0.89 and 0.95 respectively.…”

Machine learning algorithm to predict mortality in patients undergoing continuous renal replacement therapy

“…Machine learning also appears to be useful in predicting outcomes of critically ill patients or patients with AKI [18][19][20][21]. However, machine learning algorithms have not been applied to patients undergoing CRRT for AKI.…”

Artificial Intelligence in the Intensive Care Unit

“…Although the algorithm was highly specific (98%), it only had a sensitivity of 26%, severely limiting its utility [29]. Other studies have been published describing the use of machine learning models in generating patient-specific risk scores for pulmonary emboli [30], risk stratification of ARDS [31], prediction of acute kidney injury in severely burned patients [32] and in general ICU populations [33], prediction of volume responsiveness after fluid administration [34] and identification of patients likely to develop complicated Clostridium difficile infection [35].…”