Context-Aware Depth and Pose Estimation for Bronchoscopic Navigation

Abstract

Endobronchial intervention is increasingly used as a minimal invasive means of lung intervention. Vision-based localization approaches are often sensitive to image artifacts in bronchoscopic videos. In this paper, a robust navigation system based on a context-aware depth recovery approach for monocular video images is presented. To handle the artifacts, a conditional generative adversarial learning framework is proposed for reliable depth recovery. The accuracy of depth estimation and camera localization is validated on an in vivo dataset. Both quantitative and qualitative results demonstrate that the depth recovered with the proposed method preserves better structural information of airway lumens in the presence of image artifacts, and the improved camera localization accuracy demonstrates its clinical potential for bronchoscopic navigation.

Generative Localization With Uncertainty Estimation Through Video-CT Data for Bronchoscopic Biopsy

“…Recently, the deep learning-based approaches significantly improve the depth estimation from monocular images. Therefore, some research [9][10] [11] employ the DNN to estimate the depth, which further can be registered to the 3D CT scan for localisation. However, generating large annotated in vivo datasets for DNN training is difficult due to ethical issues and the labour-intensive labelling process, so some research [9][10] [12] try to train the DNN through generated synthetic images.…”

“…Our previous work [11] proposes a context-aware depth recovery approach through a CycleGan-like network trained using unpaired videos and CT depth maps. The camera pose is estimated through maximising the similarity between predicted video depth and CT depth.…”

“…To empower the network to learn the transformation of an individual input to the desired output domain from unpaired data, the cycle consistency loss L cyc in the original CycleGAN [21] is adopted to reduce the space of possible mapping functions, which makes the learned mapping function cycle-consistent. Moreover, a consecutive warping loss L warp in our previous work [11] is incorporated to provide a spatial transformation constrain between two adjacent frames of both real and virtual images. The overall loss L of the domain transfer learning is formulated as,…”

Coupled Real-Synthetic Domain Adaptation for Real-World Deep Depth Enhancement

“…a) Depth estimation: Depth estimation refers to inferring depth information directly from monocular/stereo/multiview images or image sequences [49], [55], [56], [3]. Several benchmarks of depth recovery task for both indoor or outdoor scenes are available (e.g., NYU-V2 1 , SUN-RGBD 2 ).…”

“…Accurate depth recovery is a pre-requisite for robotic navigation and manipulation [1], [2], surgical guidance [3], and human motion analysis [4]. Currently, sensing technologies supporting commercial depth cameras are mainly based on stereo correspondence, structured lighting, time-of-flight, or a combination of these techniques [5].…”