Learning laparoscopic video shot classification for gynecological surgery

Abstract

Videos of endoscopic surgery are used for education of medical experts, analysis in medical research, and documentation for everyday clinical life. Hand-crafted image descriptors lack the capabilities of a semantic classification of surgical actions and video shots of anatomical structures. In this work, we investigate how well single-frame convolutional neural networks (CNN) for semantic shot classification in gynecologic surgery work. Together with medical experts, we manually annotate hours of raw endoscopic gynecologic surgery videos showing endometriosis treatment and myoma resection of over 100 patients. The cleaned ground truth dataset comprises 9 h of annotated video material (from 111 different recordings). We use the well-known CNN architectures AlexNet and GoogLeNet and train these architectures for both, surgical actions and anatomy, from scratch. Furthermore, we extract high-level features from AlexNet with weights from a pre-trained model from the Caffe model zoo and feed them to an SVM classifier. Our evaluation shows that we reach an average recall of .697 and .515 for classification of anatomical structures and surgical actions respectively using off-the-shelf CNN features. Using GoogLeNet, we achieve a mean recall of .782 and .617 for classification of anatomical structures and surgical actions respectively. With AlexNet the achieved recall is .615 for anatomical structures and .469 for surgical action classification respectively. The main conclusion of our work is that advances in general image classification methods transfer to the domain of endoscopic surgery videos in gynecology. This is relevant as this domain is different from natural images, e.g. it is distinguished by smoke, reflections, or a limited amount of colors.

Future-State Predicting LSTM for Early Surgery Type Recognition

“…While the aforementioned methods on video classification have achieved state-of-the-art performance on benchmark video recognition datasets like UCF-101 and Sports-1M, work on surgical video recognition has received little attention primarily due to the dearth of large-scale datasets in this domain. Previous works in surgical video analysis have addressed problems like surgical phase recognition [14], surgical gesture classification [15], tool tracking [16], classification of anatomical structures and surgical actions from video shots [17]. The most relevant to this work is the paper by Twinanda et al [4] in which they propose a pipeline for classification of the type of Laparoscopic video, which consists of frame rejection, feature extraction, feature encoding and classification.…”

Video retrieval in laparoscopic video recordings with dynamic content descriptors

“…With the comparison of adjacent color histograms and thresholds for significant motion changes, they are able to detect such keypoint moments in laparoscopic surgeries. Content classification has also been addressed recently by Petscharnig and Schoeffmann [37,38], who evaluate well-known convolutional neural network architectures for the purpose of semantic segment annotation.…”

DC‐AL GAN: Pseudoprogression and true tumor progression of glioblastoma multiform image classification based on DCGAN and AlexNet

“…AlexNet is an architecture based on CNNs that has proven successful in scene classification tasks. It is recognized as an excellent basic level, automatic scene classification technology . While a typical CNN pooling process is non‐overlapping, AlexNet, dose in fact have an overlapping pooling process.…”

“…It is recognized as an excellent basic level, automatic scene classification technology. 53,54 While a typical CNN pooling process is non-overlapping, AlexNet, dose in fact have an overlapping pooling process. This contributes to a higher classification accuracy because more original information is retained.…”