Purpose
Learning is markedly improved with high-quality feedback, yet assuring the quality of feedback is difficult to achieve at scale. Natural language processing (NLP) algorithms may be useful in this context as they can automatically classify large volumes of narrative data. However, it is unknown if NLP models can accurately evaluate surgical trainee feedback. This study evaluated which NLP techniques best classify the quality of surgical trainee formative feedback recorded as part of a workplace assessment.
Method
During the 2016–2017 academic year, the SIMPL (Society for Improving Medical Professional Learning) app was used to record operative performance narrative feedback for residents at 3 university-based general surgery residency training programs. Feedback comments were collected for a sample of residents representing all 5 postgraduate year levels and coded for quality. In May 2019, the coded comments were then used to train NLP models to automatically classify the quality of feedback across 4 categories (effective, mediocre, ineffective, or other). Models included support vector machines (SVM), logistic regression, gradient boosted trees, naive Bayes, and random forests. The primary outcome was mean classification accuracy.
Results
The authors manually coded the quality of 600 recorded feedback comments. Those data were used to train NLP models to automatically classify the quality of feedback across 4 categories. The NLP model using an SVM algorithm yielded a maximum mean accuracy of 0.64 (standard deviation, 0.01). When the classification task was modified to distinguish only high-quality vs low-quality feedback, maximum mean accuracy was 0.83, again with SVM.
Conclusions
To the authors’ knowledge, this is the first study to examine the use of NLP for classifying feedback quality. SVM NLP models demonstrated the ability to automatically classify the quality of surgical trainee evaluations. Larger training datasets would likely further increase accuracy.
Background: Cirrhosis is associated with increased perioperative risks related to hepatic decompensation. However, data are lacking regarding the incidence and outcomes of postoperative hepatic encephalopathy (HE).Objective: To determine the incidence of HE postoperatively, factors associated with its development, and its association with in-hospital mortality.Methods: Retrospective cohort study of 583 patients with cirrhosis undergoing non-hepatic surgery over a 10-year period. Outcomes included postoperative HE and in-hospital mortality and were, respectively, evaluated using multi-state modeling and Fine-Gray competing risk regression (with postoperative HE as a time-varying covariate).Results: Overall, the median Model for End-Stage Liver Disease Sodium was 10, 61.7% had a history of ascites, 49.9% esophageal varices, and 34.6% HE. The most common surgeries including abdominal/non-bowel (33.3%), orthopedic (18.0%), and bowel (12.2%). A total of 42 (7.2%) patients developed HE postoperatively during admission. The cumulative risk of HE was 7.2%, which was most associated with a history of HE, ASA class, postoperative AKI, and postoperative infection. In-hospital mortality occurred in 34 (5.8%) individuals. Only ASA class was independently associated (HR 2.46, 95%CI 1.21-5.02), but there was a trend for postoperative HE (HR 1.71, 95%CI 0.73-3.98). Discussion: HE is an uncommon but not rare postoperative complication that increases the risk of patient harm. This study implies its development is predictable.Consequently, at-risk patients should have consultation with a hepatologist before undergoing elective surgery.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.