Endoscopic Video Retrieval: A Signature-Based Approach for Linking Endoscopic Images with Video Segments

Beecks, Christian; Schoeffmann, Klaus; Lux, Mathias; Uysal, Merih Seran; Seidl, Thomas

doi:10.1109/ism.2015.21

Cited by 20 publications

(20 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As a side-effect, the recoded surgery videos benefit the surgeons' work, as they provide a great basis for documentation, training of young surgeons, and medical research. Prior work supporting these aims has been conducted by our research group in the sector of endoscopic video analysis, such as a subjective quality assessment for the impact of compression on the perceived semantic quality [13], instrument classification in laparoscopic videos [17], or extraction and linking of endoscopic key-frames to videos [3,23]. In this work, we restrict ourselves to a very specific field in minimally invasive surgery in the context of gynecology.…”

Section: Introductionmentioning

confidence: 99%

Learning laparoscopic video shot classification for gynecological surgery

Petscharnig

Schöffmann

2017

Multimed Tools Appl

View full text Add to dashboard Cite

Videos of endoscopic surgery are used for education of medical experts, analysis in medical research, and documentation for everyday clinical life. Hand-crafted image descriptors lack the capabilities of a semantic classification of surgical actions and video shots of anatomical structures. In this work, we investigate how well single-frame convolutional neural networks (CNN) for semantic shot classification in gynecologic surgery work. Together with medical experts, we manually annotate hours of raw endoscopic gynecologic surgery videos showing endometriosis treatment and myoma resection of over 100 patients. The cleaned ground truth dataset comprises 9 h of annotated video material (from 111 different recordings). We use the well-known CNN architectures AlexNet and GoogLeNet and train these architectures for both, surgical actions and anatomy, from scratch. Furthermore, we extract high-level features from AlexNet with weights from a pre-trained model from the Caffe model zoo and feed them to an SVM classifier. Our evaluation shows that we reach an average recall of .697 and .515 for classification of anatomical structures and surgical actions respectively using off-the-shelf CNN features. Using GoogLeNet, we achieve a mean recall of .782 and .617 for classification of anatomical structures and surgical actions respectively. With AlexNet the achieved recall is .615 for anatomical structures and .469 for surgical action classification respectively. The main conclusion of our work is that advances in general image classification methods transfer to the domain of endoscopic surgery videos in gynecology. This is relevant as this domain is different from natural images, e.g. it is distinguished by smoke, reflections, or a limited amount of colors.

show abstract

Section: Introductionmentioning

confidence: 99%

Learning laparoscopic video shot classification for gynecological surgery

Petscharnig

Schöffmann

2017

Multimed Tools Appl

View full text Add to dashboard Cite

show abstract

“…In fact, the semantics of different content classes can only be completely understood by having medical expert knowledge. Therefore, automatic content-based similarity search is a very challenging task in this domain and does not work well with common static content descriptors [8].…”

Section: Datasetmentioning

confidence: 99%

“…-CNN Features (CNN A ) extracted from AlexNet [27] -CNN Features (CNN G ) extracted from GoogLeNet [46] -Feature Signatures (FS) [7,8] The first two descriptors are the so-called CNN Features, also known as neural codes. These are activation weights of the last fully-connected layer in a deep convolutional neural network.…”

Section: Comparison To Static Content Descriptorsmentioning

confidence: 99%

“…similarity search) in images extracted from medical videos [8]. Feature Signatures are used to describe the visual content of images individually.…”

Section: Comparison To Static Content Descriptorsmentioning

confidence: 99%

“…Next, we propose a novel dynamic video content descriptor (MIDD) that clearly outperforms other recently proposed dynamic descriptors in both terms of retrieval performance and runtime complexity. Furthermore, we investigate whether a dynamic extension of the Feature Signatures descriptor achieve equal or better performance than static Feature Signatures, which has shown good performance for content-based retrieval in the medical domain [8]. Finally, to the best of our knowledge we are the first to investigate content similarity search for video segments in the domain of gynecologic laparoscopy.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Video retrieval in laparoscopic video recordings with dynamic content descriptors

Schoeffmann

Husslein

Kletz

et al. 2017

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

In the domain of gynecologic surgery an increasing number of surgeries are performed in a minimally invasive manner. These laparoscopic surgeries require specific psychomotor skills of the operating surgeon, which are difficult to learn and teach. This is the reason why an increasing number of surgeons promote checking video recordings of laparoscopic surgeries for the occurrence of technical errors with surgical actions. This manual surgical quality assessment (SQA) process, however, is very cumbersome and timeconsuming when carried out without any support from content-based video retrieval. Appl (2018) 77:16813-16832 Descriptor) that can be effectively used to find similar segments in a laparoscopic video database and thereby help surgeons to more quickly inspect other instances of a given error scene. We evaluate the retrieval performance of MIDD with surgical actions from gynecologic surgery in direct comparison to several other dynamic content descriptors. We show that the MIDD descriptor significantly outperforms the state-of-the-art in terms of retrieval performance as well as in terms of runtime performance. Additionally, we release the manually created video dataset of 16 classes of surgical actions from medical laparoscopy to the public, for further evaluations.

show abstract

Video content analysis of surgical procedures

Loukas

2017

Surg Endosc

View full text Add to dashboard Cite

show abstract

Endoscopic Video Retrieval: A Signature-Based Approach for Linking Endoscopic Images with Video Segments

Cited by 20 publications

References 29 publications

Learning laparoscopic video shot classification for gynecological surgery

Learning laparoscopic video shot classification for gynecological surgery

Video retrieval in laparoscopic video recordings with dynamic content descriptors

Video content analysis of surgical procedures

Contact Info

Product

Resources

About