Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure

Hovgaard, Lisette Hvid; Andersen, Steven Arild Wuyts; Konge, Lars; Dalsgaard, Tage; Larsen, Claus

doi:10.1007/s00464-018-6165-5

Cited by 19 publications

(29 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Automated assessment and feedback for self‐directed training is found out of the box in many VR surgical simulators. However, most built‐in metrics and scoring systems provided by the developers have little evidence, and subsequent scientific validation of metrics‐based assessment and feedback often is needed . In temporal bone surgery, performance metrics in VR simulation of mastoidectomy have been part of the research agenda since the early development of such systems .…”

Section: Introductionmentioning

confidence: 99%

“…However, most built-in metrics and scoring systems provided by the developers have little evidence, and subsequent scientific validation of metrics-based assessment and feedback often is needed. [10][11][12] In temporal bone surgery, performance metrics in VR simulation of mastoidectomy have been part of the research agenda since the early development of such systems. 13 Metrics and scoring systems have been further refined in several of the available VR temporal bone surgical simulators, and studies have investigated various aspects of validity evidence of metrics for automated assessment and feedback.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Expert sampling of VR simulator metrics for automated assessment of mastoidectomy performance

Andersen

Mikkelsen

Sørensen³

2019

The Laryngoscope

Self Cite

View full text Add to dashboard Cite

Objective Often the assessment of mastoidectomy performance requires time‐consuming manual rating. Virtual reality (VR) simulators offer potentially useful automated assessment and feedback but should be supported by validity evidence. We aimed to investigate simulator metrics for automated assessment based on the expert performance approach, comparison with an established assessment tool, and the consequences of standard setting. Methods The performances of 11 experienced otosurgeons and 37 otorhinolaryngology residents. Participants performed three mastoidectomies in the Visible Ear Simulator. Nine residents contributed additional data on repeated practice in the simulator. One hundred and twenty‐nine different performance metrics were collected by the simulator and final‐product files were saved. These final products were analyzed using a modified Welling Scale by two blinded raters. Results Seventeen metrics could discriminate between resident and experienced surgeons’ performances. These metrics mainly expressed various aspects of efficiency: Experts demonstrated more goal‐directed behavior and less hesitancy, used less time, and selected large and sharp burrs more often. The combined metrics‐based score (MBS) demonstrated significant discriminative ability between experienced surgeons and residents with a mean difference of 16.4% (95% confidence interval [12.6–20.2], P << 0.001). A pass/fail score of 83.6% was established. The MBS correlated poorly with the final‐product score but excellently with the final‐product score per time. Conclusion The MBS mainly reflected efficiency components of the mastoidectomy procedure, and although it could have some uses in self‐directed training, it fails to measure and encourage safe routines. Supplemental approaches and feedback are therefore required in VR simulation training of mastoidectomy. Level of Evidence 2b Laryngoscope, 129:2170–2177, 2019

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Expert sampling of VR simulator metrics for automated assessment of mastoidectomy performance

Andersen

Mikkelsen

Sørensen³

2019

The Laryngoscope

Self Cite

View full text Add to dashboard Cite

show abstract

“…The RobotiX Virtual Reality robot assisted surgery simulator was used for the robot assisted group in a standard supplied setup (3D-systems Inc., Cleveland OH) ( Fig. 2) [18][19][20][21][22][23][24][25]. This setup consists of a main console for the trainee and a tower component, containing the supplied computer and task control.…”

Section: Robotix Robot Assisted Surgical Simulatormentioning

confidence: 99%

Robot assisted versus laparoscopic suturing learning curve in a simulated setting

Leijte¹,

Blaauw²,

Workum

et al. 2019

Surg Endosc

View full text Add to dashboard Cite

Background Compared to conventional laparoscopy, robot assisted surgery is expected to have most potential in difficult areas and demanding technical skills like minimally invasive suturing. This study was performed to identify the differences in the learning curves of laparoscopic versus robot assisted suturing. Method Novice participants performed three suturing tasks on the EoSim laparoscopic augmented reality simulator or the RobotiX robot assisted virtual reality simulator. Each participant performed an intracorporeal suturing task, a tilted plane needle transfer task and an anastomosis needle transfer task. To complete the learning curve, all tasks were repeated up to twenty repetitions or until a time plateau was reached. Clinically relevant and comparable parameters regarding time, movements and safety were recorded. Intracorporeal suturing time and cumulative sum analysis was used to compare the learning curves and phases. Results Seventeen participants completed the learning curve laparoscopically and 30 robot assisted. Median first knot suturing time was 611 s (s) for laparoscopic versus 251 s for robot assisted (p < 0.001), and this was 324 s versus 165 (sixth knot, p < 0.001) and 257 s and 149 s (eleventh knot, p < 0.001) respectively on base of the found learning phases. The percentage of 'adequate surgical knots' was higher in the laparoscopic than in the robot assisted group. First knot: 71% versus 60%, sixth knot: 100% versus 83%, and eleventh knot: 100% versus 73%. When assessing the 'instrument out of view' parameter, the robot assisted group scored a median of 0% after repetition four. In the laparoscopic group, the instrument out of view increased from 3.1 to 3.9% (left) and from 3.0 to 4.1% (right) between the first and eleventh knot (p > 0.05). Conclusion The learning curve of minimally invasive suturing shows a shorter task time curve using robotic assistance compared to the laparoscopic curve. However, laparoscopic outcomes show good end results with rapid outcome improvement.

show abstract

“…Many studies have determined that most aspects of the modules are realistic and useful for training, proving the face and content validity, respectively . The validity of fully robotic procedures such as radical prostatectomy, thoracic lobectomy, and vaginal cuff closure has also been proven. However, the concurrent validity of the RXM has not been investigated before.…”

Section: Introductionmentioning

confidence: 99%

Concurrent, face, content, and construct validity of the RobotiX Mentor simulator for robotic basic skills

Alshuaibi

Perrenot

Hubert

et al. 2020

Robotics Computer Surgery

View full text Add to dashboard Cite

Objectives: To assess several criteria, such as concurrent, face, content, and construct validity of the RobotiX Mentor (RXM) simulator for basic robotic skills and to compare virtual and actual dry lab dome.Methods: A prospective study was conducted from December 2017 to May 2018 using RXM and a da Vinci Si robot. 37 subjects, divided into three groups according to their initial surgical training (expert, intermediate, and novice), were evaluated in terms of six representative exercises of basic robotic specific skills as recommended by the fundamentals of robotic surgery.Results: There was a correlation between the automatic data from the RXM and the subjective evaluation with the robot. The face and content validity, which were evaluated by the experts, were generally considered high (71.5% and 62.5%, respectively). Three levels (analysis of variance [ANOVA]; P = .01) and two levels (P = .001) of experience were clearly identified by the simulator. Conclusion:Our study proves the concurrent validity and confirms the face, content, and construct validity of the RXM. K E Y W O R D S Da Vinci robot, fundamentals of robotic surgery, robotic surgery, RobotiX Mentor, simulator, surgical training

show abstract

Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure

Abstract: Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.

Cited by 19 publications

References 41 publications

Expert sampling of VR simulator metrics for automated assessment of mastoidectomy performance

Expert sampling of VR simulator metrics for automated assessment of mastoidectomy performance

Robot assisted versus laparoscopic suturing learning curve in a simulated setting

Concurrent, face, content, and construct validity of the RobotiX Mentor simulator for robotic basic skills

Contact Info

Product

Resources

About