Visually Navigated Bronchoscopy using three cycle-Consistent generative adversarial network for depth estimation

Abstract

Background-Electromagnetically Navigated Bronchoscopy (ENB) is currently the state-of-the art diagnostic and interventional bronchoscopy. CT-to-body divergence is a critical hurdle in ENB, causing navigation error and ultimately limiting the clinical efficacy of diagnosis and treatment. In this study, Visually Navigated Bronchoscopy (VNB) is proposed to address the aforementioned issue of CT-to-body divergence.Materials and Methods-We extended and validated an unsupervised learning method to generate a depth map directly from bronchoscopic images using a Three Cycle-Consistent Generative Adversarial Network (3cGAN) and registering the depth map to preprocedural CTs. We tested the working hypothesis that the proposed VNB can be integrated to the navigated bronchoscopic system based on 3D Slicer, and accurately register bronchoscopic images to pre-procedural CTs to navigate transbronchial biopsies. The quantitative metrics to asses the hypothesis we set was Absolute Tracking Error (ATE) of the tracking and the Target Registration Error (TRE) of the total navigation system. We validated our method on phantoms produced from the pre-procedural CTs of five patients who underwent ENB and on two ex-vivo pig lung specimens.Results-The ATE using 3cGAN was 6.2 +/− 2.9 [mm]. The ATE of 3cGAN was statistically significantly lower than that of cGAN, particularly in the trachea and lobar bronchus (p < 0.001). The TRE of the proposed method had a range of 11.7 to 40.5 [mm]. The TRE computed by 3cGAN was statistically significantly smaller than those computed by cGAN in two of the five cases enrolled (p < 0.05).Conclusion-VNB, using 3cGAN to generate the depth maps was technically and clinically feasible. While the accuracy of tracking by cGAN was acceptable, the TRE warrants further investigation and improvement.

Diagnostic Performance of Electromagnetic Navigation versus Virtual Navigation Bronchoscopy-Guided Biopsy for Pulmonary Lesions in a Single Institution: Potential Role of Artificial Intelligence for Navigation Planning

“…Navigation bronchoscopy has been implemented increasingly by thoracic surgeons in preoperative marking procedures, followed by sublobar resection [ 25 ]. An AI model only requires video bronchoscopic or CT images without any additional examinations [ 34 ]. Moreover, the predicted anatomical location and possible POE can be overlaid in the virtual road map.…”

BronchoPose: an analysis of data and model configuration for vision-based bronchoscopy pose estimation

“…Year Image Size Tracking Type PE (mm) AE (º) CTF (%) (Bricault et al, 1998) 1998 100x100 Local/Global 2 5 - (Mori et al, 2001) 2001 -Local --79 (Helferty and Higgins, 2002) 2002 -Local --- (Mori et al, 2002) 2002 410x410 Local --73.37 (Deligianni et al, 2004) 2004 454x487 Local 3 ± 2.26 2.18 ± 1.63 - (Nagao et al, 2004) 2004 -Local --77.79 (Shinohara et al, 2006) (Banach et al, 2021) 2021 Local 6.2 ± 2.9 --Table 1: Comparison among bronchoscopic tracking studies with regards to data and evaluation characteristics. Notably, none of the methods share a dataset (currently there is no publicly available dataset for this task) or publish their code.…”

“…Moreover, temporal learning techniques have recently been applied to other endoscopic modalites (Turan et al, 2017), but has not been appropriately tested in bronchoscopy. Additionally, depth information has lately been extensively used to improve tracking (Recasens et al, 2021;Banach et al, 2021;Shen et al, 2019;Liu et al, 2020), mixing it with generative neural networks (Zhao et al, 2020;Shen et al, 2019;Liu et al, 2020;Banach et al, 2021).…”

Haptics-Enabled Forceps with Multi-Modal Force Sensing: Towards Task-Autonomous Robotic Surgery

“…3) Camera and Target: The camera for the sensing module should have a compact form factor, sufficient resolution, and surgery compatibility. For the prototype, we chose OVM6946 (Ominivision Inc., USA), which has been adopted in many medical and surgical applications [38], [39]. It is 1mm in width and 2.27mm in length and has a 400×400 resolution.…”