Deep monocular 3D reconstruction for assisted navigation in bronchoscopy

Visentini-Scarzanella, Marco; Sugiura, T.; Kaneko, T.; Koto, Shinichiro

doi:10.1007/s11548-017-1609-2

Cited by 72 publications

(51 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The geometry differences between lungs, including the position and orientation of the airways, seem to influence the algorithms' performance. One of the most surprising results was that training AirwayNet on simulated images outperformed the models trained on camera images 14 (p<0.01). This suggests the benefit of training on the lung-specific geometry outweighs the benefit of training on up to 10 different lung geometries; however, this result is not independent from the image domain gap.…”

Section: Resultsmentioning

confidence: 99%

OffsetNet: Deep Learning for Localization in the Lung using Rendered Images

Sganga

Eng

Graetzel

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

Lung cancer is the leading cause of cancer-related death worldwide, and early diagnosis is critical to improving patient outcomes. To diagnose cancer, a highly trained pulmonologist must navigate a flexible bronchoscope deep into the branched structure of the lung for biopsy. The biopsy fails to sample the target tissue in 26-33% of cases largely because of poor registration with the preoperative CT map. We developed two deep learning approaches to localize the bronchoscope in the preoperative CT map in real time and tested the algorithms across 13 trajectories in a lung phantom and 68 trajectories in 11 human cadaver lungs. In the lung phantom, we observe performance reaching 95% precision and recall of visible airways and 3 mm average position error. On a successful cadaver lung sequence, the algorithms trained on simulation alone achieved 77%-94% precision and recall of visible airways and 4-6 mm average position error. We also compare the effect of GAN-stylizing images and we look at aggregate statistics over the entire set of trajectories.One Sentence Summary: Neural networks trained on simulated data can track a bronchoscope's movement through a plastic lung phantom and a human cadaver lung.

show abstract

Section: Resultsmentioning

confidence: 99%

OffsetNet: Deep Learning for Localization in the Lung using Rendered Images

Sganga

Eng

Graetzel

et al. 2019

2019 International Conference on Robotics and Automation (ICRA)

View full text Add to dashboard Cite

show abstract

“…The idea to use a transformer network to translate a real endoscopic image into a synthetic-like virtual image has been assessed before with the overall aim of obtaining a reconstructed topography [8,9]. We focus on the opposite transformation, synthesizing intraoperative images from real training procedures on patient-specific silicone models.…”

Section: Discussionmentioning

confidence: 99%

Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

et al. 2019

View full text Add to dashboard Cite

Current 'dry lab' surgical phantom simulators are a valuable tool for surgeons which allows them to improve their dexterity and skill with surgical instruments. These phantoms mimic the haptic and shape of organs of interest, but lack a realistic visual appearance. In this work, we present an innovative application in which representations learned from real intraoperative endoscopic sequences are transferred to a surgical phantom scenario. The term hyperrealism is introduced in this field, which we regard as a novel subform of surgical augmented reality for approaches that involve real-time object transfigurations. For related tasks in the computer vision community, unpaired cycle-consistent Generative Adversarial Networks (GANs) have shown excellent results on still RGB images. Though, application of this approach to continuous video frames can result in flickering, which turned out to be especially prominent for this application. Therefore, we propose an extension of cycle-consistent GANs, named tempCycleGAN, to improve temporal consistency. The novel method is evaluated on captures of a silicone phantom for training endoscopic reconstructive mitral valve procedures. Synthesized videos show highly realistic results with regard to 1) replacement of the silicone appearance of the phantom valve by intraoperative tissue texture, while 2) explicitly keeping crucial features in the scene, such as instruments, sutures and prostheses. Compared to the original CycleGAN approach, tempCycleGAN efficiently removes flickering between frames. The overall approach is expected to change the future design of surgical training simulators since the generated sequences clearly demonstrate the feasibility to enable a considerably more realistic training experience for minimally-invasive procedures.

show abstract

“…The CRF loss layer in the network is replaced with least squares regression. However, we do not claim this to be a direct comparison with [33] or other works because the data used is drastically different and their setup does not use of super-pixels.…”

Section: Comparative Analysismentioning

confidence: 92%

“…It is not possible to compare our results directly with existing endoscopy depth estimation work because of the [33]. This comparison allows us to judge the benefit of using a graphical model.…”

Section: Comparative Analysismentioning

confidence: 99%

Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy

Mahmood

Durr

2018

Medical Image Analysis

118

View full text Add to dashboard Cite

Colorectal cancer is the fourth leading cause of cancer deaths worldwide and the second leading cause in the United States. The risk of colorectal cancer can be mitigated by the identification and removal of premalignant lesions through optical colonoscopy. Unfortunately, conventional colonoscopy misses more than 20% of the polyps that should be removed, due in part to poor contrast of lesion topography. Imaging depth and tissue topography during a colonoscopy is difficult because of the size constraints of the endoscope and the deforming mucosa. Most existing methods make unrealistic assumptions which limits accuracy and sensitivity. In this paper, we present a method that avoids these restrictions, using a joint deep convolutional neural network-conditional random field (CNN-CRF) framework for monocular endoscopy depth estimation. Estimated depth is used to reconstruct the topography of the surface of the colon from a single image. We train the unary and pairwise potential functions of a CRF in a CNN on synthetic data, generated by developing an endoscope camera model and rendering over 200,000 images of an anatomically-realistic colon.We validate our approach with real endoscopy images from a porcine colon, transferred to a synthetic-like domain via adversarial training, with ground truth from registered computed tomography measurements. The CNN-CRF approach estimates depths with a relative error of 0.152 for synthetic endoscopy images and 0.242 for real endoscopy images. We show that the estimated depth maps can be used for reconstructing the topography of the mucosa from conventional colonoscopy images. This approach can easily be integrated into existing endoscopy systems and provides a foundation for improving computer-aided detection algorithms for detection, segmentation and classification of lesions.

show abstract

Deep monocular 3D reconstruction for assisted navigation in bronchoscopy

Abstract: The proposed method shows a novel architecture to perform real-time monocular depth estimation without losing patient specificity in bronchoscopy. Future work will include integration within SLAM systems and collection of in vivo datasets.

Cited by 72 publications

References 29 publications

OffsetNet: Deep Learning for Localization in the Lung using Rendered Images

OffsetNet: Deep Learning for Localization in the Lung using Rendered Images

Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries

Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy

Contact Info

Product

Resources

About