Comparing clinical judgment with the MySurgeryRisk algorithm for preoperative risk assessment: A pilot usability study

Abstract

Background: Major postoperative complications are associated with increased short and longterm mortality, increased healthcare cost, and adverse long-term consequences. The large amount of data contained in the electronic health record (EHR) creates barriers for physicians to recognize patients most at risk. We hypothesize, if presented in an optimal format, information from data-driven predictive risk algorithms for postoperative complications can improve physician risk assessment.Methods: Prospective, non-randomized, interventional pilot study of twenty perioperative physicians at a quarterly academic medical center. Using 150 clinical cases we compared physicians' risk assessment before and after interaction with MySurgeryRisk, a validated machine-learning algorithm predicting preoperative risk for six major postoperative complications using EHR data. Results:The area under the curve (AUC) of MySurgeryRisk algorithm ranged between 0.73 and 0.85 and was significantly higher than physicians' risk assessments (AUC between 0.47 and 0.69) for all postoperative complications except cardiovascular complications. The AUC for repeated physician's risk assessment improved by 2% to 5% for all complications with the exception of thirty-day mortality. Physicians' risk assessment for acute kidney injury and intensive care unit admission longer than 48 hours significantly improved after knowledge exchange, resulting in net reclassification improvement of 12.4% and 16%, respectively. Conclusions:The validated MySurgeryRisk algorithm predicted postoperative complications with equal or higher accuracy than pilot cohort of physicians using available clinical preoperative data. The interaction with algorithm significantly improved physicians' risk assessment.

Added Value of Intraoperative Data for Predicting Postoperative Complications: The MySurgeryRisk PostOp Extension

“…In a prospective study of the original MySurgeryRisk platform, the algorithm predicted postoperative complications with greater accuracy than physicians, but there was room for continued improvement. 19 The present study demonstrates that incorporation of intraoperative physiological time-series data improves predictive accuracy, discrimination, and precision, presumably by representing important intraoperative events and physiological changes that influence postoperative clinical trajectories and complications. Dziadzko et al 20 used a random forest model to predict mortality or the need for greater than 48 h of MV using EHR data from patients admitted to academic hospitals, achieving excellent discrimination (AUROC, 0.90), similar to MySurgeryRisk discrimination for MV for greater than 48 h (AUROC, 0.96) using both preoperative and intraoperative data.…”

<p>The Increasing Role of Artificial Intelligence in Health Care: Will Robots Replace Doctors in the Future?</p>

“…Computer vision (CV) is another emerging application of AI which involves image processing, pattern recognition, and response. 3,20,21 It is useful in several medical submit your manuscript | www.dovepress.com…”

“…14 In another study, scientists compared clinical judgment with the MySurgeryRisk algorithm for preoperative risk assessment; the authors reported the performance of this algorithm was significantly better than that of physicians who were making initial risk assessments. 20 All these pieces of evidence suggest that AI is a useful tool in enhancing the functionality of clinicians, rather than being a substitute for a human doctor. [10][11][12]30,31 It must be acknowledged that the process of integrating AI with health care practices also poses challenges such as reduced employment opportunities for humans and the liability of errors.…”

Mysteries, Epistemological Modesty, and Artificial Intelligence in Surgery

“…For some tasks, like predicting mortality among heart failure patients, logistic regression can perform as well or better than certain machine learning methods like regression tree analysis ( Austin et al, 2010 ). For complex tasks like predicting several postoperative complications, artificial intelligence models outperform regression-based techniques and clinician judgement ( Bertsimas et al, 2018 ; Bihorac et al, 2018 ; Brennan et al, 2019 ). Bertsimas et al (2018) developed an Optimal Classification Trees machine learning model to predict mortality and 18 complications following emergency surgery, demonstrating superior accuracy compared with the ACS NSQIP calculator (AUROC 0.92 vs. 0.90).…”