Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification

Liu, Daochang; Jiang, Tingting

doi:10.1007/978-3-030-00937-3_29

Cited by 53 publications

(57 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The real kinematic data were then replaced with virtual kinematic signals estimated from the video data through a spatial CNN, addressing the situation where kinematic information is available during . c: Gesture recognition with reinforcement learning [43]. At each time stamp, predictions are generated over K frames ahead.…”

Section: Conditional Random Fieldsmentioning

confidence: 99%

Gesture Recognition in Robotic Surgery: A Review

Amsterdam

Clarkson²,

Stoyanov³

2021

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. Methods: An article search was performed on 5 bibliographic databases with combinations of the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. Results: A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. Conclusion: The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. Significance: This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field.

show abstract

Section: Conditional Random Fieldsmentioning

confidence: 99%

Gesture Recognition in Robotic Surgery: A Review

Amsterdam

Clarkson²,

Stoyanov³

2021

IEEE Trans. Biomed. Eng.

View full text Add to dashboard Cite

show abstract

“…The JIG-SAWS dataset has extended annotations at a sub-task level. AI techniques learn patterns and temporal interconnections of the sub-task sequences from combinations of robot kinematics and surgical video and detect and temporally localize each sub-task [19][20][21][22][23][24]. Recently, AI models for activity recognition have been developed and tested on annotated datasets from real cases of robotic-assisted radical prostatectomy and ocular microsurgery [18][19][20].…”

Section: Surgical Phase Recognitionmentioning

confidence: 99%

Computer Vision in the Surgical Operating Room

et al. 2020

View full text Add to dashboard Cite

Background: Multiple types of surgical cameras are used in modern surgical practice and provide a rich visual signal that is used by surgeons to visualize the clinical site and make clinical decisions. This signal can also be used by artificial intelligence (AI) methods to provide support in identifying instruments, structures, or activities both in real-time during procedures and postoperatively for analytics and understanding of surgical processes. Summary: In this paper, we provide a succinct perspective on the use of AI and especially computer vision to power solutions for the surgical operating room (OR). The synergy between data availability and technical advances in computational power and AI methodology has led to rapid developments in the field and promising advances. Key Messages: With the increasing availability of surgical video sources and the convergence of technologiesaround video storage, processing, and understanding, we believe clinical solutions and products leveraging vision are going to become an important component of modern surgical capabilities. However, both technical and clinical challenges remain to be overcome to efficiently make use of vision-based approaches into the clinic.

show abstract

“…Their model is trainable with weak annotations that needs 3D bounding boxes for all instances and full voxel annotations for only a small fractions of instances. Liu et al [164] employed a novel deep reinforcement learning approach for the segmentation and classification of surgical gesture. Their approach performs well on JIGSAW dataset in terms of edit score as compared to previous similar works.…”

Section: Miscellaneousmentioning

confidence: 99%

Going Deep in Medical Image Analysis: Concepts, Methods, Challenges, and Future Directions

et al. 2019

View full text Add to dashboard Cite

Medical Image Analysis is currently experiencing a paradigm shift due to Deep Learning. This technology has recently attracted so much interest of the Medical Imaging community that it led to a specialized conference in 'Medical Imaging with Deep Learning' in the year 2018. This article surveys the recent developments in this direction, and provides a critical review of the related major aspects. We organize the reviewed literature according to the underlying Pattern Recognition tasks, and further sub-categorize it following a taxonomy based on human anatomy. This article does not assume prior knowledge of Deep Learning and makes a significant contribution in explaining the core Deep Learning concepts to the non-experts in the Medical community. Unique to this study is the Computer Vision/Machine Learning perspective taken on the advances of Deep Learning in Medical Imaging. This enables us to single out 'lack of appropriately annotated large-scale datasets' as the core challenge (among other challenges) in this research direction. We draw on the insights from the sister research fields of Computer Vision, Pattern Recognition and Machine Learning etc.; where the techniques of dealing with such challenges have already matured, to provide promising directions for the Medical Imaging community to fully harness Deep Learning in the future.

show abstract

Deep Reinforcement Learning for Surgical Gesture Segmentation and Classification

Cited by 53 publications

References 15 publications

Gesture Recognition in Robotic Surgery: A Review

Gesture Recognition in Robotic Surgery: A Review

Computer Vision in the Surgical Operating Room

Going Deep in Medical Image Analysis: Concepts, Methods, Challenges, and Future Directions

Contact Info

Product

Resources

About