Providing metrics and performance feedback in a surgical simulator

Abstract

One of the most important advantages of computer simulators for surgical training is the opportunity they afford for independent learning. However, if the simulator does not provide useful instructional feedback to the user, this advantage is significantly blunted by the need for an instructor to supervise and tutor the trainee while using the simulator. Thus, the incorporation of relevant, intuitive metrics is essential to the development of efficient simulators. Equally as important is the presentation of such metrics to the user in such a way so as to provide constructive feedback that facilitates independent learning and improvement. This paper presents a number of novel metrics for the automated evaluation of surgical technique. The general approach was to take criteria that are intuitive to surgeons and develop ways to quantify them in a simulator. Although many of the concepts behind these metrics have wide application throughout surgery, they have been implemented specifically in the context of a simulation of mastoidectomy. First, the visuohaptic simulator itself is described, followed by the details of a wide variety of metrics designed to assess the user's performance. We present mechanisms for presenting visualizations and other feedback based on these metrics during a virtual procedure. We further describe a novel performance evaluation console that displays metric-based information during an automated debriefing session. Finally, the results of several user studies are reported, providing some preliminary validation of the simulator, the metrics, and the feedback mechanisms. Several machine learning algorithms, including Hidden Markov Models and a Naïve Bayes Classifier, are applied to our simulator data to automatically differentiate users' expertise levels.

Learning Curves of Virtual Mastoidectomy in Distributed and Massed Practice

“…irtual reality (VR) simulation training is increasingly being used in surgical training, including in temporal bone surgery where evidence of the efficacy and validity of VR simulation training of novices is emerging. [1][2][3][4][5][6] Virtual reality simulation training can address some of the concerns and constraints of traditional dissection training and supervised surgery, for example, regarding patient safety issues, reduced working hours, and costs of facilities and instructors.…”

Towards continuous improvement of endoscopy standards: validation of a colonoscopy assessment form

“…We intend to use it to assess trainees in a study examining the benefit of a structured curriculum incorporating simulation on the initial learning curve of trainee endoscopists. It has also proved to be a useful training tool in clinical practice, providing trainees with proximate feedback after colonoscopy to avail of the positive effects of such feedback on learning [19,20].…”

Expert-Based Speed-Precision Control in Early Simulator Training for Novice Surgeons

“…Artificial intelligence provides well-suited concepts for reinforcement learning procedures and allows building prior benchmark knowledge on expert performance into simulator training most effectively. Specific control procedures [11] based on metric-based benchmark criteria for automatized performance comparisons, leading to decisions of the "if then" kind without supervision, not only enable the generation of learning curves in individual performance at the end of sessions but also, more importantly, enable trial-by-trial feed-back at any given moment in time during training to help individuals reach optimal performance as swiftly as possible [13][14][15][16][17][18][19][20][21]. This constitutes a far more effective approach compared with merely assessing end-of-session performance status, or differences between users after training.…”