Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks

DiPietro, Robert; Ahmidi, Narges; Malpani, Anand; Waldram, Madeleine M.; Lee, Gyusung I.; Lee, Mija R.; Vedula, S. Swaroop; Hager, Gregory D.

doi:10.1007/s11548-019-01953-x

Cited by 53 publications

(30 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When 7 labeled trials are used for training, the generative model again yields the lowest error rate (17.6%). For reference, the state-of-the-art LSTM result using 35 labeled trials is 15.3% [6]. Figure 6 shows results for edit distance; the same general trends hold.…”

Section: Experimental Designmentioning

confidence: 66%

“…As input, we use 14 kinematic signals: velocities along the 3 axes, angular velocities along the 3 axes, and gripper angle, all for both the left and right hands. All signals are provided at 50Hz, which following [6,8] we downsample by a factor of 6. The data was collected in a benchtop training environment.…”

Section: Datasetsmentioning

confidence: 99%

“…The recognition model from [6] consists of bidirectional LSTM with 3 layers, each with 64 hidden units, and is optimized using Adam and a learning rate of 10 −2.5 . All hyperparameters are carried over unchanged except the batch size: we use a batch size of 1 because it is the only possible option in many of our experiments.…”

Section: Experimental Designmentioning

confidence: 99%

“…Significant progress has been made in surgical activity recognition, especially within the context of simulated training [1,6,8], an important part of current training curricula [14]. Though promising, these approaches have relied on large amounts of annotated data, which, unlike the surgical-motion data itself, must be provided manually by experts.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Automated Surgical Activity Recognition with One Labeled Sequence

DiPietro

Hager

2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Prior work has demonstrated the feasibility of automated activity recognition in robot-assisted surgery from motion data. However, these efforts have assumed the availability of a large number of denselyannotated sequences, which must be provided manually by experts. This process is tedious, expensive, and error-prone. In this paper, we present the first analysis under the assumption of scarce annotations, where as little as one annotated sequence is available for training. We demonstrate feasibility of automated recognition in this challenging setting, and we show that learning representations in an unsupervised fashion, before the recognition phase, leads to significant gains in performance. In addition, our paper poses a new challenge to the community: how much further can we push performance in this important yet relatively unexplored regime?

show abstract

Section: Experimental Designmentioning

confidence: 66%

Section: Datasetsmentioning

confidence: 99%

Section: Experimental Designmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Automated Surgical Activity Recognition with One Labeled Sequence

DiPietro

Hager

2019

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…The Transition State Clustering (TSC) and Gaussian Mixture Model methods provide unsupervised or weakly-supervised methods for surgical trajectory segmentation [17,18]. More recently, deep learning methods have come to define the state-of-the-art, such as Temporal Convolutional Networks (TCN) [19], Time Delay Neural Network (TDNN) [7], and Long-Short Term Memory (LSTM) [6,20]. Instead of using robot kinematics data, vision-based methods have been developed based on Convolutional Neural Networks (CNN).…”

Section: Introductionmentioning

confidence: 99%

Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

Qin

Pedram

Feyzabadi

et al. 2020

2020 IEEE International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Many tasks in robot-assisted surgeries (RAS) can be represented by finite-state machines (FSMs), where each state represents either an action (such as picking up a needle) or an observation (such as bleeding). A crucial step towards the automation of such surgical tasks is the temporal perception of the current surgical scene, which requires a real-time estimation of the states in the FSMs. The objective of this work is to estimate the current state of the surgical task based on the actions performed or events occurred as the task progresses. We propose Fusion-KVE, a unified surgical state estimation model that incorporates multiple data sources including the Kinematics, Vision, and system Events. Additionally, we examine the strengths and weaknesses of different state estimation models in segmenting states with different representative features or levels of granularity. We evaluate our model on the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS), as well as a more complex dataset involving robotic intra-operative ultrasound (RIOUS) imaging, created using the da Vinci R Xi surgical system. Our model achieves a superior frame-wise state estimation accuracy up to 89.4%, which improves the state-of-the-art surgical state estimation models in both JIGSAWS suturing dataset and our RIOUS dataset.

show abstract

Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

Nwoye

González

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Segmenting and classifying activities in robot-assisted surgery with recurrent neural networks

Cited by 53 publications

References 29 publications

Automated Surgical Activity Recognition with One Labeled Sequence

Automated Surgical Activity Recognition with One Labeled Sequence

Temporal Segmentation of Surgical Sub-tasks through Deep Learning with Multiple Data Sources

Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

Contact Info

Product

Resources

About