Expert sampling of VR simulator metrics for automated assessment of mastoidectomy performance

Abstract

Objective Often the assessment of mastoidectomy performance requires time‐consuming manual rating. Virtual reality (VR) simulators offer potentially useful automated assessment and feedback but should be supported by validity evidence. We aimed to investigate simulator metrics for automated assessment based on the expert performance approach, comparison with an established assessment tool, and the consequences of standard setting. Methods The performances of 11 experienced otosurgeons and 37 otorhinolaryngology residents. Participants performed three mastoidectomies in the Visible Ear Simulator. Nine residents contributed additional data on repeated practice in the simulator. One hundred and twenty‐nine different performance metrics were collected by the simulator and final‐product files were saved. These final products were analyzed using a modified Welling Scale by two blinded raters. Results Seventeen metrics could discriminate between resident and experienced surgeons’ performances. These metrics mainly expressed various aspects of efficiency: Experts demonstrated more goal‐directed behavior and less hesitancy, used less time, and selected large and sharp burrs more often. The combined metrics‐based score (MBS) demonstrated significant discriminative ability between experienced surgeons and residents with a mean difference of 16.4% (95% confidence interval [12.6–20.2], P << 0.001). A pass/fail score of 83.6% was established. The MBS correlated poorly with the final‐product score but excellently with the final‐product score per time. Conclusion The MBS mainly reflected efficiency components of the mastoidectomy procedure, and although it could have some uses in self‐directed training, it fails to measure and encourage safe routines. Supplemental approaches and feedback are therefore required in VR simulation training of mastoidectomy. Level of Evidence 2b Laryngoscope, 129:2170–2177, 2019

The Impact of Virtual Reality Training on the Quality of Real Antromastoidectomy Performance

“…Fortunately, a new option—VR temporal bone training—has been gaining importance for the past few years, and nowadays, VR is a well-established and useful adjunct to traditional cadaveric dissection of temporal bone for trainees [ 6 , 19 , 26 ]. Such training may play an important and increasing role in education in the future, but under certain conditions and according to some standards [ 14 ]. This way of training is reportedly used in many leading training departments in Europe, and most of the remaining departments expect to implement VR simulation for temporal bone training into their residency programs in the near future [ 1 ].…”

“…The key components of the training are to avoid exceeding the anatomical boundaries of a complete mastoidectomy and to avoid violating vital structures [ 6 ]. Simulation-based technical skills training must present a good balance between automatic scoring, drilling time and self-assessment [ 14 ]. A simulator-integrated tutor function has significantly improved performance in these areas [ 6 ], and the number of objective assessment tools for self-directed practice has been reported [ 15 ].…”

“…A simulator-integrated tutor function has significantly improved performance in these areas [ 6 ], and the number of objective assessment tools for self-directed practice has been reported [ 15 ]. A feasible alternative is final-product analysis or a simulator-based automated assessment and feedback scoring system [ 14 , 15 ]. For a combined metrics-based score (MBS), a significant discriminative ability between experienced surgeons and residents was demonstrated; nevertheless, it failed to measure or encourage safe routines [ 14 ].…”

“…A feasible alternative is final-product analysis or a simulator-based automated assessment and feedback scoring system [ 14 , 15 ]. For a combined metrics-based score (MBS), a significant discriminative ability between experienced surgeons and residents was demonstrated; nevertheless, it failed to measure or encourage safe routines [ 14 ]. Moreover, at present, the automated feedback based on metrics in VR simulation does not have a sufficient empirical basis and has not been generally accepted for use in training or certification [ 16 ].…”

Standard Setting of Competency in Mastoidectomy for the Cross-Institutional Mastoidectomy Assessment Tool

“…For final-product assessment of performances in a virtual reality simulator, a cut-off of 19.5/26 points was determined based on the expert performance approach. 16 This has two major limitations: first of all, final-product analysis reflects only the end product and does not consider process. Secondly, the use of overall score does not reveal major shortcomings on some item conditions that are absolutely necessary for acceptability: for example, major violation of the facial nerve is inacceptable in the OR but represents for the specific final-product tool only a 1-point deduction from the 26-point total.…”