Binary convolutional neural network features off-the-shelf for image to video linking in endoscopic multimedia databases

Abstract

With a rigorous long-term archival of endoscopic surgeries, vast amounts of video and image data accumulate. Surgeons are not able to spend their valuable time to manually search within endoscopic multimedia databases (EMDBs) or manually maintain links to interesting sections in order to quickly retrieve relevant surgery sections. Enabling the surgeons to quickly access the relevant surgery scenes, we utilize the fact that surgeons record external images additionally to the surgery video and aim to link them to the appropriate video sequence in the EMDB using a query-by-example approach. We propose binary Convolutional Neural Network (CNN) features off-the-shelf and compare them to several baselines: pixel-based comparison (PSNR), image structure comparison (SSIM), hand-crafted global features (CEDD and feature signatures), as well as CNN baselines Histograms of Class Confidences (HoCC) and Neural Codes (NC). For evaluation, we use 5.5 h of endoscopic video material and 69 query images selected by medical experts and compare the performance of the aforementioned image mathing methods in terms of video hit rate and distance to the true playback time stamp (PTS) for correct video predictions. Our evaluation shows that binary CNN features are compact, yet powerful image descriptors for retrieval in the endoscopic imaging domain. They are able to maintain state-of-the-art performance, while providing the benefit of low storage space requirements and hence provide the best compromise.

Evolving robotic surgery training and improving patient safety, with the integration of novel technologies

“…With maturity of AI systems a number of other capabilities have also been explored like the estimation of the remaining procedural time form the real-time video feed [ 59 ] or automatic image to video retrieval [ 60 ] or potentially risk estimation [ 61 ]. An important area of work that has received limited attention at present is how such AI powered technologies for understanding surgical process and performance can be utilised in situ with effective user interfaces that appropriately support clinical workflow.…”

Live Laparoscopic Video Retrieval with Compressed Uncertainty

“…Looking at laparoscopic surgery specifically, true video retrieval works are almost non-existent: the one example available comes from research efforts parallel to ours Wang et al (2022), with a video hashing method separating motion and background for retrieving clips from the Cholec 80 dataset. Intra-video task boundary retrieval as done by Twinanda et al (2014), and frame attribution as featured in Funke et al (2018); Petscharnig and Schöffmann (2018) are the closest related work otherwise.…”

“…Early work in this area consisted of a few studies involving handcrafted features and relatively small amounts of data (Droueche et al, 2014;Amanat et al, 2018). Other tasks explored similar concepts for visual queries in surgical video content (Twinanda et al, 2014;Funke et al, 2018;Petscharnig and Schöffmann, 2018). Long after those, the one major study comes from Wang et al (2022) as a research effort parallel to ours.…”

Surgical Phase Recognition of Short Video Shots based on Temporal Modeling of Deep Features

“…Feature extraction was based on transfer learning using 'off-the-shelf' features extracted from four state-of-the-art CNNs: Alexnet, VGG19, GoogleNet, and Resnet101. These network architectures were chosen as they are known to perform well on surgical endoscopy images [3]. Transfer learning implies that the CNNs were pretrained, in this case on the Imagenet database which contains millions of natural images distributed in 1000 classes.…”

“…Laparoscopic surgery (LS), a common type of minimally invasive surgery (MIS), provides not only substantial therapeutic benefits for the patient, but also the opportunity to record the video of the operation for reasons such as documentation, technique evaluation, skills assessment, and cognitive training of junior surgeons [1], [2]. However, a major technological challenge is the effective content management of the recorded videos, given that an operation may last for more than an hour, whereas the duration of yearly operations per surgeon may exceed 1,000 hours [3].…”