Whole Stomach 3D Reconstruction and Frame Localization From Monocular Endoscope Video

Abstract

Gastric endoscopy is a common clinical practice that enables medical doctors to diagnose various lesions inside a stomach. In order to identify the location of a gastric lesion such as early cancer and a peptic ulcer within the stomach, this work addresses to reconstruct the color-textured 3D model of a whole stomach from a standard monocular endoscope video and localize any selected video frame to the 3D model. We examine how to enable structure-from-motion (SfM) to reconstruct the whole shape of a stomach from endoscope images, which is a challenging task due to the texture-less nature of the stomach surface. We specifically investigate the combined effect of chromo-endoscopy and color channel selection on SfM to increase the number of feature points. We also design a plane fitting-based algorithm for 3D point outliers removal to improve the 3D model quality. We show that whole stomach 3D reconstruction can be achieved (more than 90% of the frames can be reconstructed) by using red channel images captured under chromo-endoscopy by spreading indigo carmine (IC) dye on the stomach surface. In experimental results, we demonstrate the reconstructed 3D models for seven subjects and the application of lesion localization and reconstruction. The methodology and results presented in this paper could offer some valuable reference to other researchers and also could be an excellent tool for gastric surgeons in various computer-aided diagnosis applications.

Evaluation of the effects of an artificial intelligence system on endoscopy quality and preliminary testing of its performance in detecting early gastric cancer: a randomized controlled trial

“…In the past few years, the performance of CNN models has been generally improved by increasing the depth and fitting parameters [24,26]. In 2016, He et al proposed the concept of residuals, and it was proved to be easier for optimization and achieved better performance with fewer parameters [27].…”

Stomach 3D Reconstruction Using Virtual Chromoendoscopic Images

“…In this work, we used exactly the same endoscope video dataset from our previous work [20] . Seven videos captured from seven subjects undergoing general gastroendoscopy procedure are included in the dataset.…”

“…This is caused by the imperfection of the color image generation by the endoscope system, which combines sequentially captured R, G, and B images to form one RGB image. The color channel misalignment causes some texture patterns to appear duplicated and disturbs the SfM pipeline (See Figure 1 in [20] ). Because of that, we used single-channel images for SfM and investigated which color channel gives the best 3D reconstruction result.…”

“…In our previous study, we tackled the drawbacks of the previous studies for lesion localization by reconstructing the color-textured 3D model of a whole stomach from an endoscope video based on an SfM pipeline [19] , [20] . Although the stomach 3D reconstruction by SfM is very challenging because of texture-less stomach surfaces as shown in Figure 1(a) , we found that the whole stomach shape can be reconstructed by using red-channel images of chromoendoscopy with indigo carmine (IC) blue dye, where the IC dye acts as an enhancement substance to bring up more textures to the stomach surface as shown in Figure 1(b) .…”

Deep learning-based anatomical site classification for upper gastrointestinal endoscopy

“…Using artificial intelligence (AI) to understand the endoscopic examination process can potentially help EGD clinicians to quickly quantify their photo-documentation that summarised a case and additionally could even support the detection and identification of diseased lesions [4,17]. The use of AI for computer-assisted endoscopy can potentially support and efficiently improve the quality of endoscopy by ensuring a complete examination, by enhancing navigation with 3D mapping [2,23] or through automated procedural analysis [24]. Deep learning-based methods are the current state-of-the-art methodology for almost all image understanding and analysis problems like semantic segmentation, image recognition and classification [6,11].…”