Pd02-01 development and Validation of an Objective Scoring Tool to Evaluate Surgical Dissection: Dissection Assessment for Robotic Technique (Dart)

Abstract

INTRODUCTION AND OBJECTIVE:There is a need for effective surgical competency assessment to ensure safe and efficient transfer of skills to training robotic surgeons. Additionally, competency assessment has important implications for credentialing and improving patient outcomes. This project aimed to develop an objective scoring tool that comprehensively evaluates surgical dissection quality.METHODS: The study was conducted in two phases. During phase 1, development, the Delphi method was used to validate content of Dissection Assessment for Robotic Technique (DART). A multi-institutional and multi-disciplinary (n[12 Urology, n[1 OBGYN, n[1 General Surgery) panel of expert surgeons systematically evaluated each element; those with content validity index (CVI) of ! 0.8 were included in the final product (Figure 1). During phase 2, validation, 10 blinded reviewers evaluated 46 de-identified videos of the pelvic lymph node and seminal vesicle dissection steps during robot-assisted radical prostatectomy. Inter-rater reliability was calculated using prevalenceadjusted and bias-adjusted kappa (PABAK). Expert and trainee surgeons were defined by prior robotic caseload >100 and 100, respectively. Mann-Whitney U test was used to compare between groups.RESULTS: In phase 1, Delphi method achieved content validity in 27/28 elements after 4 rounds. The element that did not reach consensus was use of a 3-point or 5-point scale (CVI[0.64). Most surgeons (93%) agreed that it is important to maximize agreement between raters. In phase 2, the 3-point scale showed greater interrater reliability for each domain as compared with the 5-point scale (PABAK!0.52, !0.38, respectively). When rated on the 3-point scale, total scores between experts and trainees exhibited a significant difference (median 17.2 (IQR 16.9-17.4) vs. 15.7 (14.8-16.3), p<0.001).CONCLUSIONS: DART is an objective and reproducible 3point surgical assessment that effectively differentiates between levels of surgical experience.

An Assessment Tool to Provide Targeted Feedback to Robotic Surgical Trainees: Development and Validation of the End-to-End Assessment of Suturing Expertise (EASE)

“…Following the precedent our team set in developing Dissection Assessment for Robotic Technique, 10 we have published the inter-rater reliability trends of our raters from initial training to final study evaluation set. From these data, we see there is a need for thorough training to see optimal EASE scoring across all domains.…”