Content-based processing and analysis of endoscopic images and videos: A survey

Abstract

In recent years, digital endoscopy has established as key technology for medical screenings and minimally invasive surgery. Since then, various research communities with manifold backgrounds have picked up on the idea of processing and automatically analyzing the inherently available video signal that is produced by the endoscopic camera. Proposed works mainly include image processing techniques, pattern recognition, machine learning methods and Computer Vision algorithms. While most contributions deal with realtime assistance at procedure time, the post-procedural processing of recorded videos is still in its infancy. Many post-processing problems are based on typical Multimedia methods like indexing, retrieval, summarization and video interaction, but have only been sparsely addressed so far for this domain. The goals of this survey are (1) to introduce this research field to a broader audience in the Multimedia community to stimulate further research, (2) to describe domain-specific characteristics of endoscopic videos that need to be addressed in a pre-processing step, and (3) to systematically bring together the very diverse research results for the first time to provide a broader overview of related research that is currently not perceived as belonging together.

Learning laparoscopic video shot classification for gynecological surgery

“…single frame, early fusion, late fusion, and slow fusion [6]. The importance of deep learning in medical image analysis and content-based processing and analysis of endoscopic images and video also is apparent from the work of Litjens et al [9] and Muenzer et al [12] respectively.…”

Video retrieval in laparoscopic video recordings with dynamic content descriptors

“…These works include (i) pre-processing of images such as image enhancement [14,41] and content filtering [2,36], (ii) real-time support at procedure time such as diagnostic decision support and computer-integrated surgery [44,45], as well as (iii) post-procedural applications such as quality/skills assessment [31,51] and contentbased retrieval [47,48]. A broad overview of such works is provided in an extensive survey by Muenzer et al [35]. While many works have been proposed for the first two categories mentioned above, content similarity search for supporting surgical quality assessment has been addressed only sparsely in the literature so far.…”

Multi‐instance multi‐label learning for surgical image annotation

“…Over the last decade, there is a growing research activity in the development of methods and systems that provide contextual information about surgical procedures, such as phase detection, instrument usage, event/task recognition, and skills performance. 1 Ultimately, the goal is to enhance the situation awareness of the operator, to improve surgical workflow, to minimize medical errors, and generally to optimize the management and quality of patient care. 2 The application of advanced data analysis algorithms, computing technologies, and imaging equipment in the operating room (OR) have played a key role towards this direction.…”