Robust camera localisation with depth reconstruction for bronchoscopic navigation

Abstract

Purpose Bronchoscopy is a standard technique for airway examination, providing a minimally invasive approach for both diagnosis and treatment of pulmonary diseases. To target lesions identified pre-operatively, it is necessary to register the location of the bronchoscope to the CT bronchial model during the examination. Existing vision-based techniques rely on the registration between virtually rendered endobronchial images and videos based on image intensity or surface geometry. However, intensity-based approaches are sensitive to illumination artefacts while gradient-based approaches are vulnerable to surface texture.Methods In this paper, depth information is employed in a novel way to achieve continuous and robust camera localisation. Surface shading has been used to recover depth from endobronchial images. The pose of the bronchoscopic camera is estimated by maximising the similarity between the depth recovered from a video image and that captured from a virtual camera projection of the CT model. The normalised cross-correlation and mutual information have both been used and compared for the similarity measure.Results The proposed depth-based tracking approach has been validated on both phantom and in vivo data. It outperforms the existing vision-based registration methods resulting in smaller pose estimation error of the bronchoscopic camera. It is shown that the proposed approach is more robust to illumination artefacts and surface texture and less sensitive to camera pose initialisation.

Context-Aware Depth and Pose Estimation for Bronchoscopic Navigation

“…Current bronchoscopic navigation systems can be broadly classified into three main categories including sensorbased [12], vision-based [6], [8], [9] and the hybrid systems [13], [14]. External tracking devices such as EM sensors can be integrated into flexible endoscopes to report real-time tip pose.…”

“…The imagebased registration approaches are more robust to respiratory motion because the scope moves with the airway simultaneously [6]. The similarity metric can be based on image intensity [6], [7], gradient [8], [14], depth [9] or airway lumen features [15]. The similarity in terms of image intensity proposed in [6], [7] requires the generation of realistic renderings from the CT airway model.…”

“…However, the surface textures of the endobronchial walls and the illumination conditions are difficult to recover. Instead of using image intensity, several attempts transform the registration into pq space [8], [14] or depth maps [9] which are more texture-independent while better preserving the morphology of airway bifurcations. Since pixel-based image registration is computationally expensive, more efficient localization methods based on airway lumen detection [15], [16] are developed for navigation at distal airways where multiple bifurcations can be observed.…”

“…The purpose of this paper is focused on the development of a vision-based localization framework. Several 2D/3D registration methods based on different similarity measures [6]- [9] have been developed for bronchoscopic navigation. Among them, the method based on depth maps [9] is shown to be more robust to illumination variations compared to other methods based on intensity or gradient features.…”

“…Several 2D/3D registration methods based on different similarity measures [6]- [9] have been developed for bronchoscopic navigation. Among them, the method based on depth maps [9] is shown to be more robust to illumination variations compared to other methods based on intensity or gradient features. However, the depth estimation from bronchoscopic images is still challenging due to the following issues: 1) conventional Shape from Motion (SFM) technique is difficult to properly recover the depth with the small field of view and the lack of salient features; 2) constrained by the small airway dimensions, only monocular bronchoscopes are widely available, making the depth recovery approaches using stereo camera settings inapplicable; 3) matched in vivo video frames and CT virtual views are difficult to obtain because of the timeconsuming and error-prone manual alignment, while supervised depth learning algorithms often require a large amount of labelled data; 4) in real clinical scenarios, image artifacts such as blurring and water bubbles as shown in Fig.…”