Recognizing Surgical Activities with Recurrent Neural Networks

DiPietro, Robert; Lea, Colin; Malpani, Anand; Ahmidi, Narges; Vedula, S. Swaroop; Lee, Gyusung I.; Lee, Mija R.; Hager, Gregory D.

doi:10.1007/978-3-319-46720-7_64

Cited by 115 publications

(106 citation statements)

References 14 publications

Supporting

Mentioning

104

Contrasting

Order By: Relevance

“…Much more discriminative features contribute to the compelling performance of phase recognition and meanwhile, alleviate the challenge of high annotation workload. DiPietro et al (2016) used RNN to model the robot kinematics and achieved an accurate phase recognition for robotic surgery. Lea et al (2016) employed temporal filters to convolve the sequential stacked spatial features extracted from CNN.…”

Section: Surgical Video Analysismentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

145

View full text Add to dashboard Cite

Surgical tool presence detection and surgical phase recognition are two fundamental yet challenging tasks in surgical video analysis and also very essential components in various applications in modern operating rooms. While these two analysis tasks are highly correlated in clinical practice as the surgical process is well-defined, most previous methods tackled them separately, without making full use of their relatedness. In this paper, we present a novel method by developing a multi-task recurrent convolutional network with correlation loss (MTRCNet-CL) to exploit their relatedness to simultaneously boost the performance of both tasks. Specifically, our proposed MTRCNet-CL model has an end-to-end architecture with two branches, which share earlier feature encoders to extract general visual features while holding respective higher layers targeting for specific tasks. Given that temporal information is crucial for phase recognition, long-short term memory (LSTM) is explored to model the sequential dependencies in the phase recognition branch. More importantly, a novel and effective correlation loss is designed to model the relatedness between tool presence and phase identification of each video frame, by minimizing the divergence of predictions from the two branches. Mutually leveraging both lowlevel feature sharing and high-level prediction correlating, our MTRCNet-CL method can encourage the interactions between the two tasks to a large extent, and hence can bring about benefits to each other. Extensive experiments on a large surgical video dataset (Cholec80) demonstrate outstanding performance of our proposed method, consistently exceeding the state-of-the-art methods by a large margin (e.g., 89.1% v.s. 81.0% for the mAP in tool presence detection and 87.4% v.s. 84.5% for F1 score in phase recognition). The code can be found on our project website.

show abstract

Section: Surgical Video Analysismentioning

confidence: 99%

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Jin

Dou

et al. 2020

Medical Image Analysis

145

View full text Add to dashboard Cite

show abstract

“…The MISTIC-SL dataset contains 49 right-handed trials of a suture throw followed by a surgeon's knot, employing in total four maneuvers: suture throw, knot tying, grasp pull run suture, and intermaneuver segment [10]. This dataset has been used to learn representations of surgical motions [11] and recognize surgical activities [10]. However, the MISTIC-SL dataset is not publicly available at the moment.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Previous supervised learning methods include the use of hidden Markov models [15] [16], conditional random fields [17], and bag of spatio-temporal features [13]. More recent methods include the use of Recurrent Neural Networks for recognizing surgical activities [10], [11]. Nonetheless, these approaches were tested using data from the same distribution as the training data, and do not account for the disparity encountered from randomized initial conditions.…”

Section: Background and Related Workmentioning

confidence: 99%

DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robots

Madapana

Low

Voyles

et al. 2019

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Datasets are an essential component for training effective machine learning models. In particular, surgical robotic datasets have been key to many advances in semiautonomous surgeries, skill assessment, and training. Simulated surgical environments can enhance the data collection process by making it faster, simpler and cheaper than real systems. In addition, combining data from multiple robotic domains can provide rich and diverse training data for transfer learning algorithms. In this paper, we present the DESK (Dexterous Surgical Skill) dataset. It comprises a set of surgical robotic skills collected during a surgical training task using three robotic platforms: the Taurus II robot, Taurus II simulated robot, and the YuMi robot. This dataset was used to test the idea of transferring knowledge across different domains (e.g. from Taurus to YuMi robot) for a surgical gesture classification task with seven gestures. We explored three different scenarios: 1) No transfer, 2) Transfer from simulated Taurus to real Taurus and 3) Transfer from Simulated Taurus to the YuMi robot. We conducted extensive experiments with three supervised learning models and provided baselines in each of these scenarios. Results show that using simulation data during training enhances the performance on the real robot where limited real data is available. In particular, we obtained an accuracy of 55% on the real Taurus data using a model that is trained only on the simulator data. Furthermore, we achieved an accuracy improvement of 34% when 3% of the real data is added into the training process.

show abstract

“…The begin and end times of tasks or sub-tasks must be automatically identified from within the entire procedure because manual identification through post-operative video review is overly time consuming and not scalable. Machine learning algorithms have been used with promising initial results in laparoscopic [8,9,10,11] and robotic-assisted surgeries [12,13,14,15,16,17,18].…”

Section: Introductionmentioning

confidence: 99%

Novel evaluation of surgical activity recognition models using task-based efficiency metrics

et al. 2019

View full text Add to dashboard Cite

Purpose: Surgical task-based metrics (rather than entire procedure metrics) can be used to improve surgeon training and, ultimately, patient care through focused training interventions. Machine learning models to automatically recognize individual tasks or activities are needed to overcome the otherwise manual effort of video review. Traditionally, these models have been evaluated using frame-level accuracy. Here, we propose evaluating surgical activity recognition models by their effect on task-based efficiency metrics. In this way, we can determine when models have achieved adequate performance for providing surgeon feedback via metrics from individual tasks. Methods: We propose a new CNN-LSTM model, RP-Net-V2, to recognize the 12 steps of robotic-assisted radical prostatectomies (RARP). We evaluated our model both in terms of conventional methods (e.g. Jaccard Index, task boundary accuracy) as well as novel ways, such as the accuracy of efficiency metrics computed from instrument movements and system events.Results: Our proposed model achieves a Jaccard Index of 0.85 thereby outperforming previous models on robotic-assisted radical prostatectomies. Additionally, we show that metrics computed from tasks automatically identified using RP-Net-V2 correlate well with metrics from tasks labeled by clinical experts. Conclusions: We demonstrate that metrics-based evaluation of surgical activity recognition models is a viable approach to determine when models can be used to quantify surgical efficiencies. We believe this approach and our results illustrate the potential for fully automated, post-operative efficiency reports.

show abstract

Recognizing Surgical Activities with Recurrent Neural Networks

Cited by 115 publications

References 14 publications

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

Multi-task recurrent convolutional network with correlation loss for surgical video analysis

DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robots

Novel evaluation of surgical activity recognition models using task-based efficiency metrics

Contact Info

Product

Resources

About