Video-based surgical skill assessment using 3D convolutional neural networks

Abstract

Purpose: A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill.Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability of a robotic surgery system to capture kinematic data. In contrast, we investigate a method for automatic, objective skill assessment that requires video data only. This has the advantage that video can be collected effortlessly during minimally invasive and robot-assisted training scenarios. Methods: Our method builds on recent advances in deep learning-based video classification. Specifically, we propose to use an inflated 3D ConvNet to classify snippets, i.e., stacks of a few consecutive frames, extracted from surgical video. The network is extended into a Temporal Segment Network during training. Results: We evaluate the method on the publicly available JIGSAWS dataset, which consists of recordings of basic robot-assisted surgery tasks performed on a dry lab bench-top model. Our approach achieves high skill classification accuracies ranging from 95.1% to 100.0%. Conclusions: Our results demonstrate the feasibility of deep learning-based assessment of technical skill from surgical video. Notably, the 3D ConvNet is able to learn meaningful patterns directly from the data, alleviating the need for man-

A Survey of Vision-Based Human Action Evaluation Methods

“…Without the provision of kinematic data or tool motion data from a robotic surgical system, Funke et al [107] first proposed a video-based surgical skill assessment method using a deep neural network. They fine-tuned a pretrained 3D ConvNet to extract spatio-temporal features from video snippets and used a temporal segment network to resolve ambiguities in single video snippets for the task of surgical skill assessment.…”

Language-based translation and prediction of surgical navigation steps for endoscopic wayfinding assistance in minimally invasive surgery

“…The usage of temporal features between endoscopic image has been validated in [13] to estimate procedure durations accurately. Furthermore, in [15], the advantage of combining spatio-temporal features for classification tasks was shown, suggesting that latent temporal information between labeled images exists. Alternatively to conventionally training models directly on image features, our approach is oriented on the prediction of time series using statistical models [12,16].…”

Motion analysis of the JHU-ISI Gesture and Skill Assessment Working Set using Robotics Video and Motion Assessment Software

“…Since data from one's own da Vinci system are unavailable to most investigators, the data in JIGSAWS are used to evaluate new models to predict surgical skill. Studies using the JIGSAWS data include an assessment of skill based on video data applied to a convolutional neural network [24], studies of holistic features of the data [25] and gesture analysis [26]. Investigators have used the JIGSAWS dataset to develop predictive models with a deep learning framework, as well as a neural network and a deep neural network which were then used to evaluate study participants [27][28][29].…”