Developing artificial intelligence models for medical student suturing and knot-tying video-based assessment and coaching

Abstract

Background Early introduction and distributed learning have been shown to improve student comfort with basic requisite suturing skills. The need for more frequent and directed feedback, however, remains an enduring concern for both remote and in-person training. A previous in-person curriculum for our second-year medical students transitioning to clerkships was adapted to an at-home video-based assessment model due to the social distancing implications of COVID-19. We aimed to develop an Artificial Intelligence (AI) model to perform video-based assessment. Methods Second-year medical students were asked to submit a video of a simple interrupted knot on a penrose drain with instrument tying technique after self-training to proficiency. Proficiency was defined as performing the task under two minutes with no critical errors. All the videos were first manually rated with a pass-fail rating and then subsequently underwent task segmentation. We developed and trained two AI models based on convolutional neural networks to identify errors (instrument holding and knot-tying) and provide automated ratings. Results A total of 229 medical student videos were reviewed (150 pass, 79 fail). Of those who failed, the critical error distribution was 15 knot-tying, 47 instrument-holding, and 17 multiple. A total of 216 videos were used to train the models after excluding the low-quality videos. A k-fold cross-validation (k = 10) was used. The accuracy of the instrument holding model was 89% with an F-1 score of 74%. For the knot-tying model, the accuracy was 91% with an F-1 score of 54%. Conclusions Medical students require assessment and directed feedback to better acquire surgical skill, but this is often time-consuming and inadequately done. AI techniques can instead be employed to perform automated surgical video analysis. Future work will optimize the current model to identify discrete errors in order to supplement video-based rating with specific feedback.

Validity evidence supporting clinical skills assessment by artificial intelligence compared with trained clinician raters

“…In addition, more evidence must be collected for both AIBA and EBA to support the use of scores to guide entrustment decisions where the relation to measures such as clinical performance should be investigated. Following previous validity reports on AI assessments, accuracy, precision, sensitivity and F1‐score were reported in this study to support the scoring inference 42,43 . However, in accordance with contemporary guidelines for trustworthy AI in medicine, 44 our study emphasises the importance of reporting validity threats in terms of robustness and explainability.…”

“…Following previous validity reports on AI assessments, accuracy, precision, sensitivity and F1-score were reported in this study to support the scoring inference. 42,43 However, in accordance with contemporary guidelines for trustworthy AI in medicine, 44 our study emphasises the importance of reporting validity threats in terms of robustness and explainability.…”

Laparoscopic Surgery: A Systematic Review of Loop and Knot Security, Varying with the Suture and Sequences, Throws, Rotation and Destabilization of Half-Knots or Half-Hitches

“…Another approach is pre-training of laparoscopic psychomotor kills [47], visual force feedback [48], and the use of a knot-tying board with the measurements of the vertical and lateral forces exerted [49]. Also noteworthy is the importance that a trainer cannot be replaced by a video [50], that telementoring and training onsite are equally effective [51], that video registration and artificial intelligence can help in the evaluation of trainees [52], and the importance of the mental image of knots [53] and of fatigue [54]. Ideally, a surgeon should demonstrate minimal skills and knowledge before operating on women, and a structural pre-training [40] and assessing the knot security of 50 to 100 knots with a dynamometer might be a straightforward and reproducible way to evaluate this.…”

What if We Replace Pass/Fail Grading with Decision Making by Machine? Toward Competency-based Medical Education

“…Two hundred sixteen suturing and knot-tying videos were used to train the models. The accuracy of the instrument holding and the knot-tying model was 89% with an F-1 score of 74% and 91% with an F-1 score of 54%, respectively (11). Generally, the accessibility and flexibility of ML methods have resulted in various successful applications in medical education and assessment (12)(13)(14)(15).…”