OffsetNet: Deep Learning for Localization in the Lung using Rendered Images

Abstract

Lung cancer is the leading cause of cancer-related death worldwide, and early diagnosis is critical to improving patient outcomes. To diagnose cancer, a highly trained pulmonologist must navigate a flexible bronchoscope deep into the branched structure of the lung for biopsy. The biopsy fails to sample the target tissue in 26-33% of cases largely because of poor registration with the preoperative CT map. We developed two deep learning approaches to localize the bronchoscope in the preoperative CT map in real time and tested the algorithms across 13 trajectories in a lung phantom and 68 trajectories in 11 human cadaver lungs. In the lung phantom, we observe performance reaching 95% precision and recall of visible airways and 3 mm average position error. On a successful cadaver lung sequence, the algorithms trained on simulation alone achieved 77%-94% precision and recall of visible airways and 4-6 mm average position error. We also compare the effect of GAN-stylizing images and we look at aggregate statistics over the entire set of trajectories.One Sentence Summary: Neural networks trained on simulated data can track a bronchoscope's movement through a plastic lung phantom and a human cadaver lung.

Generative Localization With Uncertainty Estimation Through Video-CT Data for Bronchoscopic Biopsy

“…Therefore, some research [9][10] [11] employ the DNN to estimate the depth, which further can be registered to the 3D CT scan for localisation. However, generating large annotated in vivo datasets for DNN training is difficult due to ethical issues and the labour-intensive labelling process, so some research [9][10] [12] try to train the DNN through generated synthetic images.…”

“…OffsetNet [12] employs DNN to regress the 6 DOF relative pose between two adjacent real and rendering images, and further accumulate them to generate the whole trajectory of camera. The performance of proposed network is improved by augmenting the training data with rendering images.…”

Intraoperative surgery room management: A deep learning perspective

“…was the most studied application, and public dataset were made available for experimentation, such as Cholec80 , 29 consisting on a collection of cholecystectomy video. Apart from abdominal surgery, in Reference 60, the authors proposed OffsetNet, a network for localization in the lung, and in Reference 61, Bouget et al worked on images from neurosurgical microscopes.…”

Autonomous Driving in the Lung using Deep Learning for Localization

“…In the lung domain, Visentini-Scarzanella et al and used a CNN to estimate the depth map of 2D images in a phantom lung, which could then be registered to the 3D map, but localization is not reported [12]. In our previous work, we used a CNN to localize a bronchoscope in real-time by predicting the offset between the camera image and a rendering at the expected position [20]. This approach, called OffsetNet, showed 1.4 mm accuracy on a phantom lung sequence, but it fails to track other sequences.…”