Task versus Subtask Surgical Skill Evaluation of Robotic Minimally Invasive Surgery

Abstract

Evaluating surgical skill is a time consuming, subjective, and difficult process. This paper compares two methods of identifying the skill level of a subject given motion data from a benchtop surgical task. In the first method, we build discrete Hidden Markov Models at the task level, and test against these models. In the second method, we build discrete Hidden Markov Models of surgical gestures, called surgemes, and evaluate skill at this level. We apply these techniques to 57 data sets collected from the da Vinci surgical system. Our current techniques have achieved accuracy levels of 100% using task level models and known gesture segmentation, 95% with task level models and unknown gesture segmentation, and 100% with the surgeme level models in correctly identifying the skill level. We observe that, although less accurate, the second method requires less prior label information. Also, the surgeme level classification provided more insights into what subjects did well, and what they did poorly.

Methods and Tools for Objective Assessment of Psychomotor Skills in Laparoscopic Surgery

“…Despite this, determining the number of states to use and the model's topology can be a daunting task, subjected in many cases to trial and error [107,112]. Moreover, reported results seem not to be so different between using MMs or HMMs [106], although HMMs fiexibility may help to better accommodate the needs of different assessment systems.…”

“…Sequential analysis of surgical tasks by Markov modeling has been the most common approach [49,50,[106][107][108][109][110][111]. Simple Markov models (MM) interpret sequences of actions as a series of steps, defined by a closed number of states and the probabilities from going from one state to another.…”

“…Simple Markov models (MM) interpret sequences of actions as a series of steps, defined by a closed number of states and the probabilities from going from one state to another. States are usually taken as sets of predefined surgical maneuvers (or surgemes) [106]. MMs that decompose tasks into states based on force/torque patterns have been employed by Rosen et al for modeling surgical steps with an 86% success rate [50].…”

“…MMs that decompose tasks into states based on force/torque patterns have been employed by Rosen et al for modeling surgical steps with an 86% success rate [50]. However, definition of these surgemes is often done manually and, as an alternative, hidden Markov models (HMM) have been widely used [49,[106][107][108][109][110][111]. Rather than requiring a previous definition of actions, HMMs model their states as probabilistic functions based on physical observations (such as motion metrics) [112].…”

Surgical Gesture Classification from Video Data

“…Most of the prior work on surgical gesture recognition (see, e.g., [4][5][6]) uses hidden Markov models (HMMs) to analyze kinematic data stored by the robot, such as the position of the robot tools, angles between robot joints, velocity measurements and force/torque signatures. All these approaches model each surgeme as one or more states of an HMM.…”