Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

show abstract

“…dissection, exposure, resection, etc.) [19]. Such consideration should allow for more clinically applicable annotations.…”

Section: Challenges In Temporal Annotationmentioning

confidence: 99%

Challenges in surgical video annotation

Ward

Fer

Ban

et al. 2021

Computer Assisted Surgery

View full text Add to dashboard Cite

show abstract

“…In an attempt to deeply analyze the surgical actions, Nwoye et al [73] presented a method for fine-grained surgical actions recognition based on modeling each phase as action triplets <instrument, verb, target> representing the tool activity. They annotated 40 videos from the Cholec80…”

Section: ) Gestures Segmentationmentioning

confidence: 99%

A Review on Deep Learning in Minimally Invasive Surgery

et al. 2021

View full text Add to dashboard Cite

In the last five years, deep learning has attracted great interest in computer-assisted systems for Minimally Invasive Surgery. The straightforward accessibility to images in surgical interventions makes deep neural networks enormously powerful for solving classification problems in complex surgical scenarios. The objective of this work is to provide readers a survey on deep learning models applied to minimally invasive surgery, identifying the different architectures used depending on the application, the results achieved until now, and the publicly available surgical datasets that can be used for validating new studies. A total of 85 publications have been extracted from manual research from four databases (IEEE Xplorer, Springer Link, Science Direct, and ACM Digital Library). After analyzing all these studies, they have been classified into four applications: surgical image analysis, surgical task analysis, surgical skill assessment, and automation of surgical tasks. This work provides a technical description of these works and a comparison among them. Finally, promising research directions to advance in this field are identified.

show abstract

“…Recognizing fine-grained hepatocystic anatomy through semantic segmentation could help surgeons better assess the critical view of safety (CVS), a universally recommended technique consisting in well exposing anatomical landmarks to prevent bile duct injuries [42]. Additionally, segmentation masks of hepatocystic structures could be leveraged by deep learning models for automatic assessment of CVS [40] and surgical action recognition [43] to improve their performance. We believe that generating a dataset for video semantic segmentation of hepatocystic anatomy will promote surgical data science research and accelerate the development of applications for surgical safety.…”

Section: Endoscapesmentioning

confidence: 99%

Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation

Alapatt¹,

Mascagni²,

Vardazaryan³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets. This bottleneck is particularly prohibitive in highly specialized and regulated fields such as medicine and surgery, where video semantic segmentation could have important applications but data and expert annotations are scarce. In these settings, temporal clues and anatomical constraints could be leveraged during training to improve performance. Here, we present Temporally Constrained Neural Networks (TCNN), a semi-supervised framework used for video semantic segmentation of surgical videos. In this work, we show that autoencoder networks can be used to efficiently provide both spatial and temporal supervisory signals to train deep learning models. We test our method on a newly introduced video dataset of laparoscopic cholecystectomy procedures, Endoscapes, and an adaptation of a public dataset of cataract surgeries, CaDIS. We demonstrate that lower-dimensional representations of predicted masks can be leveraged to provide a consistent improvement on both sparsely labeled datasets with no additional computational cost at inference time. Further, the TCNN framework is model-agnostic and can be used in conjunction with other model design choices with minimal additional complexity.

show abstract

Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

Cited by 59 publications

References 24 publications

Challenges in surgical video annotation

Challenges in surgical video annotation

A Review on Deep Learning in Minimally Invasive Surgery

Temporally Constrained Neural Networks (TCNN): A framework for semi-supervised video semantic segmentation

Contact Info

Product

Resources

About