Content-based retrieval in videos from laparoscopic surgery

Abstract

In the field of medical endoscopy more and more surgeons are changing over to record and store videos of their endoscopic procedures for long-term archival. These endoscopic videos are a good source of information for explanations to patients and follow-up operations. As the endoscope is the "eye of the surgeon", the video shows the same information the surgeon has seen during the operation, and can describe the situation inside the patient much more precisely than an operation report would do. Recorded endoscopic videos can also be used for training young surgeons and in some countries the long-term archival of video recordings from endoscopic procedures is even enforced by law. A major challenge, however, is to efficiently access these very large video archives for later purposes. One problem, for example, is to locate specific images in the videos that show important situations, which are additionally captured as static images during the procedure. This work addresses this problem and focuses on contentbased video retrieval in data from laparoscopic surgery. We propose to use feature signatures, which can appropriately and concisely describe the content of laparoscopic images, and show that by using this content descriptor with an appropriate metric, we are able to efficiently perform content-based retrieval in laparoscopic videos. In a dataset with 600 captured static images from 33 hours recordings, we are able to find the correct video segment for more than 88% of these images.Feature signatures, as for instance investigated in the work of Beecks, 1 are a compact yet efficient way of describing the content of an image. They are obtained by clustering characteristic image features into a compact feature representation that adapts to individual image contents. In this paper, we propose to utilize feature signatures over a 7-dimensional feature space including position, color, and texture information for content-based retrieval in laparoscopic videos. To this end, we investigate different distance-based similarity measures including the Earth Mover's Distance, 2 the Signature Quadratic Form Distance, 3, 4 and the Signature Matching Distance. 5 Through a large evaluation with video data from 33 hours of laparoscopic surgery we show that the proposed signature-based approaches are able to efficiently perform content-based retrieval in laparoscopic videos. Description of PurposeIn endoscopic surgery, operating surgeons usually record video segments and capture static images during the operation, which they often need after the procedure. 6 However, while the static images can be used for later discussions with patients and colleagues, and can be easily integrated into operation reports, they lack of a complete and precise information, such as movements of operation instruments and the dynamics of performed actions. The recorded videos, on the other hand, contain all these important details, since they show exactly the same images that were visible on the monitors in the operation room, which were also used by the...

Binary convolutional neural network features off-the-shelf for image to video linking in endoscopic multimedia databases

“…This work is novel, as to the best of our knowledge, we are the first to provide such a statistically motivated binarization scheme for CNN features off-the-shelf. Moreover, this work extends the work of Carlos et al [7], Beecks et al [3], and Schöffmann et al [27] which also tackle this problem for the domain of endoscopic videos. In contrast to these works, we do not only evaluate whether the video was hit, but we also the correct frame position.…”

“…The problem of image to video linking for endoscopic surgeries (strictly speaking: retrieval of a video from an endoscopic video database based on a query image) has already been topic to recent research [3,7,27] which are the works most related to our research. In Carlos et al [7], retrieval based on global features (Color and Edge Directivity Descriptor CEDD [9], auto color correlogram [16], and pyramid histogram of oriented gradients PHOG [5]) fusion methods by rank and score as well as a localized version of CEDD using the SIMPLE approach [17] on every fifth video frame is evaluated.…”

“…We assume that the automatic creation of bookmarks -linking the externally captured images to the correct videos and playback time stamps (PTS) -saves the surgeons' time at tasks such as education or documentation, thus enabling them to spend more time on tasks such as medical research. It is important to note that in practice there is no way for a trivial interlinking of captured images and recorded videos (as also described in related work [3,7,27]). This problem originates from the use of different systems with distinct encoders.…”

“…Of course, the main advantage of such approaches is the computational efficiency: we have to calculate the descriptors for the EMDB only once, followed by a computational cheap distance calculation -while on the other hand, we have to calculate PSNR or SSIM against every frame in the EMDB. As image descriptors, we evaluate CEDD [9] and feature signatures, since these descriptors have shown good performance in related work [3,7,27]. Recent literature also shows that CNN features provide an astounding baseline for visual recognition [26].…”

Video Retrieval System for Meniscal Surgery to Improve Health Care Services

“…Weight is assigned to each feature describing the potential to be used as a feature signature representative. For comparison between two signatures, earth mover's distance, signature quadratic form distance, and signature matching distance are used in combination [5]. To retrieve endoscopy videos from repository, feature signature is computed based on global features, that is, colour, position, and texture.…”