How to Bring Surgery to the Next Level: Interpretable Skills Assessment in Robotic-Assisted Surgery

Abstract

<b><i>Introduction:</i></b> A surgeon’s technical skills are an important factor in delivering optimal patient care. Most existing methods to estimate technical skills remain subjective and resource intensive. Robotic-assisted surgery (RAS) provides a unique opportunity to develop objective metrics using key elements of intraoperative surgeon behavior which can be captured unobtrusively, such as instrument positions and button presses. Recent studies have shown that objective metrics based on these data (referred to as objective performance indicators [OPIs]) correlate to select clinical outcomes during robotic-assisted radical prostatectomy. However, the current OPIs remain difficult to interpret directly and, therefore, to use within structured feedback to improve surgical efficiencies. <b><i>Methods:</i></b> We analyzed kinematic and event data from da Vinci surgical systems (Intuitive Surgical, Inc., Sunnyvale, CA, USA) to calculate values that can summarize the use of robotic instruments, referred to as OPIs. These indicators were mapped to broader technical skill categories of established training protocols. A data-driven approach was then applied to further sub-select OPIs that distinguish skill for each technical skill category within each training task. This subset of OPIs was used to build a set of logistic regression classifiers that predict the probability of expertise in that skill to identify targeted improvement and practice. The final, proposed feedback using OPIs was based on the coefficients of the logistic regression model to highlight specific actions that can be taken to improve. <b><i>Results:</i></b> We determine that for the majority of skills, only a small subset of OPIs (2–10) are required to achieve the highest model accuracies (80–95%) for estimating technical skills within clinical-like tasks on a porcine model. The majority of the skill models have similar accuracy as models predicting overall expertise for a task (80–98%). Skill models can divide a prediction into interpretable categories for simpler, targeted feedback. <b><i>Conclusion:</i></b> We define and validate a methodology to create interpretable metrics for key technical skills during clinical-like tasks when performing RAS. Using this framework for evaluating technical skills, we believe that surgical trainees can better understand both what can be improved and how to improve.

Predicting Surgical Experience After Robotic Nerve-sparing Radical Prostatectomy Simulation Using a Machine Learning–based Multimodal Analysis of Objective Performance Metrics

“…Prior work using OPIs has focused on quantitative assessment of individual surgical skills (ie, suturing and dissection) 13,21 or assessing the potential for vascular injury during resident training. 22 One notable study used deep learning to predict continence recovery after RARP using the same type of performance metrics.…”

“…12 Objective performance indicators (OPIs) are metrics objectively derived from kinematic and event-based data collected from the da Vinci robotic system (Intuitive Surgical). 13 OPIs were validated with video analysis using GEARS during nerve-sparing RARP, highlighting skill differences between various expert surgeons while correlating with clinical outcomes. 14 Deeper into the hierarchical analysis of technical performance are surgical gestures, representing actions via instruments during a surgical step, which have been used to quantify suturing performance during vesicoureteral anastomosis and in predicting urinary continence recovery.…”

Parallel, component training in robotic total mesorectal excision

“…Alternative ways of assessing robotic pro ciency include using cumulative sum analysis (CUSUM) learning curves, however, we felt using this approach would be di cult in assessing and attributing individual robotic component pro ciency with regards to operative and clinical outcomes. The use of objective performance indicators derived directly from the robotic system, reporting kinematic and event data, will provide the most objective assessment of technical pro ciency in robotic surgery (25). These automated performance metrics will provide accurate and timely assessment data, which will transform the assessment of robotic competency as the utility of these metrics becomes more widespread.…”

Identification of Main Influencers of Surgical Efficiency and Variability Using Task-Level Objective Metrics: A Five-Year Robotic Sleeve Gastrectomy Case Series

“…Finally, task-based objective performance indicators (OPIs) other than total operative time are often neglected despite offering the potential for improved and focused feedback ( 14 – 17 ). There exists an opportunity to develop more objective methods that can scale for broad use ( 18 – 20 ) given a limited number of studies use subjective methods to estimate the impact of a surgeon's technical skills on patient outcomes ( 21 – 25 ). Additionally, these objective methods need to be able to be applied to an individual surgeon, within institutions, or across institutions.…”