Multi-center validation of machine learning model for preoperative prediction of postoperative mortality

Abstract

Accurate prediction of postoperative mortality is important for not only successful postoperative patient care but also for information-based shared decision-making with patients and efficient allocation of medical resources. This study aimed to create a machine-learning prediction model for 30-day mortality after a non-cardiac surgery that adapts to the manageable amount of clinical information as input features and is validated against multi-centered rather than single-centered data. Data were collected from 454,404 patients over 18 years of age who underwent non-cardiac surgeries from four independent institutions. We performed a retrospective analysis of the retrieved data. Only 12–18 clinical variables were used for model training. Logistic regression, random forest classifier, extreme gradient boosting (XGBoost), and deep neural network methods were applied to compare the prediction performances. To reduce overfitting and create a robust model, bootstrapping and grid search with tenfold cross-validation were performed. The XGBoost method in Seoul National University Hospital (SNUH) data delivers the best performance in terms of the area under receiver operating characteristic curve (AUROC) (0.9376) and the area under the precision-recall curve (0.1593). The predictive performance was the best when the SNUH model was validated with Ewha Womans University Medical Center data (AUROC, 0.941). Preoperative albumin, prothrombin time, and age were the most important features in the model for each hospital. It is possible to create a robust artificial intelligence prediction model applicable to multiple institutions through a light predictive model using only minimal preoperative information that can be automatically extracted from each hospital.

Artificial Intelligence–enabled Decision Support in Surgery

“…Among these 5 articles, 4 [26][27][28]35 had wide confidence intervals ranging from 0.15 to 0.35, and 1, 37 which used waveform input features, had a narrow confidence interval of 0.01 (albeit with moderate performance in predicting postoperative deterioration, AUROC 0.71). Among all 36 articles, 5 31,33,35,39,40 (13.8%) performed external validation, 2 9,29 (5.6%) performed real-time validation, and 29 (80.6%) performed internal validation only. None of the articles described equity analyses in which model performance was stratified by sex or race.…”

“…Twentythree articles 9,19,21,[23][24][25]27,29,30,[32][33][34]36,37,39,41,42,[44][45][46][48][49][50] (63.9%) reported precision metrics (area under the precision-recall curve, positive predictive value, or F1 score). Twenty-five articles 9,20,21,[23][24][25][26][27][28]31,33,34,36,38,40,[42][43][44][45][46][47][48][49][50] (69.4%) included explainability mechanisms to convey the relative importance of input features in determining outputs. Thirteen articles 9,16,17,…”

“…Twenty-three articles9,19,21,23–25,27,29,30,32–34,36,37,39,41,42,44–46,48–50 (63.9%) reported precision metrics (area under the precision-recall curve, positive predictive value, or F1 score). Twenty-five articles9,20,21,23–28,31,33,34,36,38,40,42–50 (69.4%) included explainability mechanisms to convey the relative importance of input features in determining outputs. Thirteen articles9,16,17,20,25,27,29,30,35,38,45,46,50 (36.1%) presented a framework that could be used for clinical implementation; none of the articles assessed the efficacy of clinical implementation.…”

Using Machine Learning and Deep Learning Algorithms to Predict Postoperative Outcomes Following Anterior Cervical Discectomy and Fusion

“…ML has a distinct advantage over conventional statistical modeling in its ability to accurately predict postoperative prognoses. 5 Leveraging ML will aid clinicians in identifying at-risk patients and help spine surgery teams appraise the risk of their patients in the preoperative setting. 6 Past ML studies have studied medical complications and health utilization in the context of ACDF.…”

“…The burden of postoperative health care use presents a potential application of machine learning (ML) techniques, which may be capable of accurately identifying at-risk patients using patient-specific predictors. ML has a distinct advantage over conventional statistical modeling in its ability to accurately predict postoperative prognoses 5. Leveraging ML will aid clinicians in identifying at-risk patients and help spine surgery teams appraise the risk of their patients in the preoperative setting 6.…”