AI in Surgical Curriculum Design and Unintended Outcomes for Technical Competencies in Simulation Training

Abstract

ImportanceTo better elucidate the role of artificial intelligence (AI) in surgical skills training requires investigations in the potential existence of a hidden curriculum.ObjectiveTo assess the pedagogical value of AI-selected technical competencies and their extended effects in surgical simulation training.Design, Setting, and ParticipantsThis cohort study was a follow-up of a randomized clinical trial conducted at the Neurosurgical Simulation and Artificial Intelligence Learning Centre at the Montreal Neurological Institute, McGill University, Montreal, Canada. Surgical performance metrics of medical students exposed to an AI-enhanced training curriculum were compared with a control group of participants who received no feedback and with expert benchmarks. Cross-sectional data were collected from January to April 2021 from medical students and from March 2015 to May 2016 from experts. This follow-up secondary analysis was conducted from June to September 2022. Participants included medical students (undergraduate year 0-2) in the intervention cohorts and neurosurgeons to establish expertise benchmarks.ExposurePerformance assessment and personalized feedback by an intelligent tutor on 4 AI-selected learning objectives during simulation training.Main Outcomes and MeasuresOutcomes of interest were unintended performance outcomes, measured by significant within-participant difference from baseline in 270 performance metrics in the intervention cohort that was not observed in the control cohort.ResultsA total of 46 medical students (median [range] age, 22 [18-27] years; 27 [59%] women) and 14 surgeons (median [range] age, 45 [35-59] years; 14 [100%] men) were included in this study, and no participant was lost to follow-up. Feedback on 4 AI-selected technical competencies was associated with additional performance change in 32 metrics over the entire procedure and 20 metrics during tumor removal that was not observed in the control group. Participants exposed to the AI-enhanced curriculum demonstrated significant improvement in safety metrics, such as reducing the rate of healthy tissue removal (mean difference, −7.05 × 10−5 [95% CI, −1.09 × 10−4 to −3.14 × 10−5] mm3 per 20 ms; P < .001) and maintaining a focused bimanual control of the operative field (mean difference in maximum instrument divergence, −4.99 [95% CI, −8.48 to −1.49] mm, P = .006) compared with the control group. However, negative unintended effects were also observed. These included a significantly lower velocity and acceleration in the dominant hand (velocity: mean difference, −0.13 [95% CI, −0.17 to −0.09] mm per 20 ms; P < .001; acceleration: mean difference, −2.25 × 10−2 [95% CI, −3.20 × 10−2 to −1.31 × 10−2] mm per 20 ms2; P < .001) and a significant reduction in the rate of tumor removal (mean difference, −4.85 × 10−5 [95% CI, −7.22 × 10−5 to −2.48 × 10−5] mm3 per 20 ms; P < .001) compared with control. These unintended outcomes diverged students’ movement and efficiency performance metrics away from the expertise benchmarks.Conclusions and RelevanceIn this cohort study of medical students, an AI-enhanced curriculum for bimanual surgical skills resulted in unintended changes that improved performance in safety but negatively affected some efficiency metrics. Incorporating AI in course design requires ongoing assessment to maintain transparency and foster evidence-based learning objectives.

Artificial intelligence in surgery

“…An exploration of the potential biases of AI explanations in surgical video assessment 122 -namely under-and overskilling biases, based on the surgeon's level of training-highlights the importance of comparisons with current gold standards and utilizing AI outputs as data with which to iterate, learn and optimize toward real-world benefits. In another example, an AI coach had both positive and negative impacts on the proficiency of medical students performing neurosurgical simulation, including improved technical performance at the expense of reduced efficiency 123 . This example also shows the importance of expert guidance in the development and implementation of AI tools in specialized domains, as well as the need for ongoing assessment of such programs.…”

Artificial Intelligence and Healthcare Simulation: The Shifting Landscape of Medical Education

“…True to form, the trainees in this AI-guided arm did much better on the four competencies than the trainees in the control that had trained without any highlighting of benchmarks. However, a global evaluation also indicated that other metrics showed significant decrements in operative performance among members of the AI group, including deficiencies in the amount of tumor resected [ 105 ]. These results should remind us that not everything associated with AI is a de facto positive outcome.…”