Implementing assessments of robot‐assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

Goldenberg, Mitchell; Lee, Jason Y.; Kwong, Jethro C.C.; Grantcharov, Teodor P.; Costello, Anthony J.

doi:10.1111/bju.14219

Cited by 24 publications

(25 citation statements)

References 104 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One study used accelerometers attached to robotic instrument arms and force‐torque sensors on task boards to show that experienced surgeons performed a variety of dry laboratory exercises with lower vibration magnitudes, lower forces and shorter completion times relative to novices . While multiple tools exist for the assessment of robotic skills, the majority of the evidence is mixed, of insufficient quality and compares groups at the extremes of skill, that is, novice and expert surgeons .…”

Section: Introductionmentioning

confidence: 99%

Experts vs super‐experts: differences in automated performance metrics and clinical outcomes for robot‐assisted radical prostatectomy

Hung

Chen

et al. 2018

BJU International

View full text Add to dashboard Cite

ObjectivesTo evaluate automated performance metrics (APMs) and clinical data of experts and super-experts for four cardinal steps of robot-assisted radical prostatectomy (RARP): bladder neck dissection; pedicle dissection; prostate apex dissection; and vesico-urethral anastomosis. Subjects and MethodsWe captured APMs (motion tracking and system events data) and synchronized surgical video during RARP. APMs were compared between two experience levels: experts (100-750 cases) and super-experts (2100-3500 cases). Clinical outcomes (peri-operative, oncological and functional) were then compared between the two groups. APMs and outcomes were analysed for 125 RARPs using multi-level mixed-effect modelling. ResultsFor the four cardinal steps selected, super-experts showed differences in select APMs compared with experts (P < 0.05). Despite similar PSA and Gleason scores, super-experts outperformed experts clinically with regard to peri-operative outcomes, with a greater lymph node yield of 22.6 vs 14.9 nodes, respectively (P < 0.01), less blood loss (125 vs 130 mL, respectively; P < 0.01), and fewer readmissions at 30 days (1% vs 13%, respectively; P = 0.02). A similar but nonsignificant trend was seen for oncological and functional outcomes, with super-experts having a lower rate of biochemical recurrence compared with experts (5% vs 15%, respectively; P = 0.13) and a higher continence rate at 3 months (36% vs 18%, respectively; P = 0.14). ConclusionWe found that experts and super-experts differed significantly in select APMs for the four cardinal steps of RARP, indicating that surgeons do continue to improve in performance even after achieving expertise. We hope ultimately to identify associations between APMs and clinical outcomes to tailor interventions to surgeons and optimize patient outcomes.

show abstract

Section: Introductionmentioning

confidence: 99%

Experts vs super‐experts: differences in automated performance metrics and clinical outcomes for robot‐assisted radical prostatectomy

Hung

Chen

et al. 2018

BJU International

View full text Add to dashboard Cite

show abstract

“…There is increasing evidence that surgeon technical skills influence postoperative patient outcomes [1][2][3][4][5][6][7]. Methods to estimate technical skills and provide feedback throughout surgeons' learning curves can lead to more efficient training [1,[8][9][10].…”

Section: Introductionmentioning

confidence: 99%

How to Bring Surgery to the Next Level: Interpretable Skills Assessment in Robotic-Assisted Surgery

et al. 2020

View full text Add to dashboard Cite

Introduction: A surgeon’s technical skills are an important factor in delivering optimal patient care. Most existing methods to estimate technical skills remain subjective and resource intensive. Robotic-assisted surgery (RAS) provides a unique opportunity to develop objective metrics using key elements of intraoperative surgeon behavior which can be captured unobtrusively, such as instrument positions and button presses. Recent studies have shown that objective metrics based on these data (referred to as objective performance indicators [OPIs]) correlate to select clinical outcomes during robotic-assisted radical prostatectomy. However, the current OPIs remain difficult to interpret directly and, therefore, to use within structured feedback to improve surgical efficiencies. Methods: We analyzed kinematic and event data from da Vinci surgical systems (Intuitive Surgical, Inc., Sunnyvale, CA, USA) to calculate values that can summarize the use of robotic instruments, referred to as OPIs. These indicators were mapped to broader technical skill categories of established training protocols. A data-driven approach was then applied to further sub-select OPIs that distinguish skill for each technical skill category within each training task. This subset of OPIs was used to build a set of logistic regression classifiers that predict the probability of expertise in that skill to identify targeted improvement and practice. The final, proposed feedback using OPIs was based on the coefficients of the logistic regression model to highlight specific actions that can be taken to improve. Results: We determine that for the majority of skills, only a small subset of OPIs (2–10) are required to achieve the highest model accuracies (80–95%) for estimating technical skills within clinical-like tasks on a porcine model. The majority of the skill models have similar accuracy as models predicting overall expertise for a task (80–98%). Skill models can divide a prediction into interpretable categories for simpler, targeted feedback. Conclusion: We define and validate a methodology to create interpretable metrics for key technical skills during clinical-like tasks when performing RAS. Using this framework for evaluating technical skills, we believe that surgical trainees can better understand both what can be improved and how to improve.

show abstract

“…[1][2][3] In general, laparoscopic and robotic surgery provides the possibility to evaluate surgeons' skills based on intracorporal surgical videos. 4,5 Analysis of these videos offers the opportunity to gain insight into past performance and to relate intraoperative events to adverse postoperative outcomes to learn for the future [6][7][8][9] .…”

Section: Introductionmentioning

confidence: 99%

“…Effective training and assessment of performance are fundamental to ensuring that surgeons reach their intended goal and operate safely with maximum preservation of functions. 7,10,11 The eld of video review and postoperative results analysis is focused on predicting postoperative results and reducing complications. 1,2,12,13 Even though multiple groups have investigated the possibility of video review it is unclear if video review has found its place in daily clinical practice.…”

Section: Introductionmentioning

confidence: 99%

Robot-Assisted Radical Prostatectomy: A Survey on the Influence of Postoperative Results Analysis and Surgical Video Review on Postoperative Complications and Functional Results.

Beulens

Veerman

Brinkman

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: Effective training and assessment of performance are fundamental to ensuring that surgeons reach their intended goal and operate safely with maximum preservation of functions. The field of video review and postoperative results analysis is focused on predicting postoperative results and reducing complications. Even though multiple groups have investigated the possibility of video review it is unclear if video review has found its place in daily clinical practice. To investigate the experiences and opinions of surgeons in the field of Robot-assisted Radical Prostatectomy (RARP) on the influence of video reviews and postoperative results analysis on postoperative complications and functional results.Design, Setting, and Participants: RARP surgeons who were expected to perform video reviews and postoperative results analysis were identified. A total of 93 RARP surgeons were invited to participate in this survey. Online questionnaires were distributed. The questionnaire contained the following domains: background information of the RARP surgeons, evaluation of the use of postoperative results analysis and surgical video review, and future recommendations.Results: A total of 30 RARP surgeons responded to the questionnaire. 27 respondents organized periodical results analyses, 17 of them reviewed edited videos as part of standard clinical practice in their hospital. Most respondents recommend video review, and are convinced it improves outcomes through self-reflection, feedback from a colleague, or from seeing different techniques and ‘tricks’. Conclusion: RARP surgeons have accepted the implementation of postoperative results analysis and surgical video review as forms of quality assurance and self-reflection. They use edited surgical videos to gain insight into the specific facets of RARP related to postoperative complications and functional outcomes.

show abstract

Implementing assessments of robot‐assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

Cited by 24 publications

References 104 publications

Experts vs super‐experts: differences in automated performance metrics and clinical outcomes for robot‐assisted radical prostatectomy

Experts vs super‐experts: differences in automated performance metrics and clinical outcomes for robot‐assisted radical prostatectomy

How to Bring Surgery to the Next Level: Interpretable Skills Assessment in Robotic-Assisted Surgery

Robot-Assisted Radical Prostatectomy: A Survey on the Influence of Postoperative Results Analysis and Surgical Video Review on Postoperative Complications and Functional Results.

Contact Info

Product

Resources

About