Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy

Abstract

Colorectal cancer is the fourth leading cause of cancer deaths worldwide and the second leading cause in the United States. The risk of colorectal cancer can be mitigated by the identification and removal of premalignant lesions through optical colonoscopy. Unfortunately, conventional colonoscopy misses more than 20% of the polyps that should be removed, due in part to poor contrast of lesion topography. Imaging depth and tissue topography during a colonoscopy is difficult because of the size constraints of the endoscope and the deforming mucosa. Most existing methods make unrealistic assumptions which limits accuracy and sensitivity. In this paper, we present a method that avoids these restrictions, using a joint deep convolutional neural network-conditional random field (CNN-CRF) framework for monocular endoscopy depth estimation. Estimated depth is used to reconstruct the topography of the surface of the colon from a single image. We train the unary and pairwise potential functions of a CRF in a CNN on synthetic data, generated by developing an endoscope camera model and rendering over 200,000 images of an anatomically-realistic colon.We validate our approach with real endoscopy images from a porcine colon, transferred to a synthetic-like domain via adversarial training, with ground truth from registered computed tomography measurements. The CNN-CRF approach estimates depths with a relative error of 0.152 for synthetic endoscopy images and 0.242 for real endoscopy images. We show that the estimated depth maps can be used for reconstructing the topography of the mucosa from conventional colonoscopy images. This approach can easily be integrated into existing endoscopy systems and provides a foundation for improving computer-aided detection algorithms for detection, segmentation and classification of lesions.

Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

“…The idea to use a transformer network to translate a real endoscopic image into a synthetic-like virtual image has been assessed before with the overall aim of obtaining a reconstructed topography [8,9]. We focus on the opposite transformation, synthesizing intraoperative images from real training procedures on patient-specific silicone models.…”

Dense Depth Estimation in Monocular Endoscopy With Self-Supervised Learning Methods

“…Mahmood et al simulate pairs of color images and dense depth maps from CT data for depth estimation network training. During the application phase, they use a Generative Adversarial Network to convert real endoscopic images to simulation-like ones and then feed them to the trained depth estimation network [10]. In their work, the appearance transformer network is trained separately by simply mimicking the appearance of simulated images but without knowledge of the target task, i. e., depth estimation, which can lead to decreased performance up to incorrect depth estimates.…”

Deep Learning in Cardiology

“…Researchers could also utilize CRFs which are graphical models that capture context-aware information and are able to incorporate higher order statistics, which traditional deep learning methods are unable to do. CRFs have been jointly trained with CNNs and have been used in depth estimation in endoscopy [269] and liver segmentation in CT [270]. There are also cardiology applications that used CRFs with deep learning as a segmentation refinement step in fundus photography [171], [174], and in LV/RV [143].…”