Real-Time Clinical Decision Support Based on Recurrent Neural Networks for In-Hospital Acute Kidney Injury: External Validation and Model Interpretation (Preprint)

Abstract

BACKGROUND Acute kidney injury(AKI) is commonly encountered in clinical practice and associated with poor patient outcomes and increased healthcare costs. AKI poses significant challenges for clinicians but effective measures for the prediction and prevention of AKI are lacking. Previously published AKI prediction models mostly have simple design without external validation. Furthermore, little is known about how to link the model output and clinical decision supports due to the blackbox nature of the neural network models. OBJECTIVE We aimed to present an externally validated recurrent neural network (RNN)-based prediction model for in-hospital AKI, and to show the explainability of the model in relation to clinical decision support. METHODS Study populations were all patients aged ≥ 18 years and hospitalized more than a week from 2013 to 2017 in two tertiary hospitals in Korea (Seoul National University Bundang Hospital and Seoul National University Hospital). All demographics, laboratory values, vital signs, and clinical conditions were obtained from the EHR of each hospital. A total of 102 variables included in the model. Each variable falls into two categories: static and dynamic variables. We developed two-stage hierarchical prediction models (model 1 and model 2) using RNN algorithms. The outcome variable for Model 1 was the occurrence of AKI within 7 days from the present. Model 2 predicts the future trajectory of Cr values up to 72 hours. Internal validation was performed by 5-fold cross validation using the training set, and then external validation was done using independent test set. RESULTS Of a total of 118,893 patients initially screened, after excluding cases with missing data and estimated glomerular filtration rate <15 ml/min/1.73m2 or end-stage kidney disease, 40,552 patients in training cohort and 4,084 in external validation cohort (test cohort) were used for model development. Model 1 with the observation window of 3 days predicts AKI development with the area under the curve of 0.80 (sensitivity 0.72, specificity 0.89) in external validation. The model 2 predicted the future creatinine values within 3 days with the mean square errors of 0.04-0.06 for patients with higher risks of AKI and 0.05-0.12 for those with lower risks. On the basis of the developed models, we showed the probability of AKI according to the feature values in total patients and each individual with partial dependence plots and individual conditional expectation plots. In addition, we estimated the effects of feature modifications such as nephrotoxic drug discontinuation on the future creatinine levels. CONCLUSIONS We developed and externally validated a real-time AKI prediction model using RNN algorithms. Our model could provide real-time assessment of future AKI occurrences and individualized risk factors for AKI in general inpatient cohorts. These suggest approaches to support clinical decisions based on the prediction models for in-hospital AKI.