Development and validation of an interpretable neural network for prediction of postoperative in-hospital mortality

Abstract

While deep neural networks (DNNs) and other machine learning models often have higher accuracy than simpler models like logistic regression (LR), they are often considered to be “black box” models and this lack of interpretability and transparency is considered a challenge for clinical adoption. In healthcare, intelligible models not only help clinicians to understand the problem and create more targeted action plans, but also help to gain the clinicians’ trust. One method of overcoming the limited interpretability of more complex models is to use Generalized Additive Models (GAMs). Standard GAMs simply model the target response as a sum of univariate models. Inspired by GAMs, the same idea can be applied to neural networks through an architecture referred to as Generalized Additive Models with Neural Networks (GAM-NNs). In this manuscript, we present the development and validation of a model applying the concept of GAM-NNs to allow for interpretability by visualizing the learned feature patterns related to risk of in-hospital mortality for patients undergoing surgery under general anesthesia. The data consists of 59,985 patients with a feature set of 46 features extracted at the end of surgery to which we added previously not included features: total anesthesia case time (1 feature); the time in minutes spent with mean arterial pressure (MAP) below 40, 45, 50, 55, 60, and 65 mmHg during surgery (6 features); and Healthcare Cost and Utilization Project (HCUP) Code Descriptions of the Primary current procedure terminology (CPT) codes (33 features) for a total of 86 features. All data were randomly split into 80% for training (n = 47,988) and 20% for testing (n = 11,997) prior to model development. Model performance was compared to a standard LR model using the same features as the GAM-NN. The data consisted of 59,985 surgical records, and the occurrence of in-hospital mortality was 0.81% in the training set and 0.72% in the testing set. The GAM-NN model with HCUP features had the highest area under the curve (AUC) 0.921 (0.895–0.95). Overall, both GAM-NN models had higher AUCs than LR models, however, had lower average precisions. The LR model without HCUP features had the highest average precision 0.217 (0.136–0.31). To assess the interpretability of the GAM-NNs, we then visualized the learned contributions of the GAM-NNs and compared against the learned contributions of the LRs for the models with HCUP features. Overall, we were able to demonstrate that our proposed generalized additive neural network (GAM-NN) architecture is able to (1) leverage a neural network’s ability to learn nonlinear patterns in the data, which is more clinically intuitive, (2) be interpreted easily, making it more clinically useful, and (3) maintain model performance as compared to previously published DNNs.

Integration of feature vectors from raw laboratory, medication and procedure names improves the precision and recall of models to predict postoperative mortality and acute kidney injury

“…The last several years have seen an explosion in the number of papers using machine learning (ML) techniques to predict a variety of perioperative outcomes. Models have been successfully developed to predict key outcomes such as hypotension 1 , 2 , mortality 3 – 6 , readmission 7 , and acute kidney injury (AKI) 4 , 8 – 11 . As a group, these papers have proven the underlying hypothesis that ML techniques can be applied to healthcare data to predict outcomes.…”

Machine learning in perioperative medicine: a systematic review

“… AUROCs of the auxiliary models for ICU AKI were 0.7537, 0.7589, 0.7950, 0.7333 and 0.7654. Not applicable Lee CK, 2021 [ 36 ] Retrospective/observational single center Prediction of mortality in post-operative patients 59,985 Post-operative mortality Generalized additive models with neural networks (GAM-NNs). AUC 0.921 Model performance was compared to a standard LR model Jeong YS, 2021 [ 37 ] Retrospective/observational single center To make a proper model for predicting postoperative major cardiac event (MACE) in ESRD patients undergoing general anesthesia.…”

Novel Insight into the Relationship Between Muscle-Fat and Bone in Type 2 Diabetes Ranging from Normal Weight to Obesity

“…The model was increasingly used in various authoritative studies in different fields including ecology, biology, and human clinical research. 17 , 18 A novel model was adopted to analyze the relationship between the combination of muscle-fat and bone health in our study. In our study, the results of GAMs suggested that the relationship of fat and muscle on BMD was not one-way linear ( Figure 3 ).…”