Bronchoscopic video synchronization for interactive multimodal inspection of bronchial lesions

Chang, Qi; Byrnes, Patrick; Ahmad, Danish; Toth, Jennifer; Bascom, Rebecca; Higgins, William E.

doi:10.1117/12.2579931

Cited by 2 publications

(8 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, Figure 3 demonstrates the advantages of our new framework as compared to our previous multimodal synchronization work. 10 The previous approach did not offer true registered views, nor did it enable fusion between the video and 3D surface model. Also, the superior performance in displaying the landmarks in our current method enables a more flexible choice of camera poses to image the airway wall, thereby facilitating more exhaustive surface coverage.…”

Section: Resultsmentioning

confidence: 99%

“…For instance, our previous multimodal synchronization method necessitated the registration of about 20 frames. 10 The methods proposed in this paper only demand extra manual alignment of a small amount number of (around 5) beside lesion frames. This minimizes user involvement and makes the whole process more achievable and practical.…”

Section: Frames Located Between the Landmark Set To Help Fill Endolum...mentioning

confidence: 99%

“…Finally, this approach dramatically improves upon our earlier multimodal video synchronization methodology, which did not align the video frames to the CT-based airway tree model. 10 The earlier work also necessitated manual physician review of the videos to identify suspect lesion frames, did not draw on landmarks associated with the airway tree, and did not have a rigorous means for selecting suitable video frames to fill gaps.…”

Section: Frames Located Between the Landmark Set To Help Fill Endolum...mentioning

confidence: 99%

“…no video frame in one source views the airways at exactly the same pose as in another source. 10 This forces the physician to manually examine each modality's video stream separately and then mentally correlate lesion observations. This manual inspection is not only extremely tedious and skill dependent, but also poses the risk of missed lesions, thereby reducing diagnostic confidence.…”

Section: Introductionmentioning

confidence: 99%

“…This manual inspection is not only extremely tedious and skill dependent, but also poses the risk of missed lesions, thereby reducing diagnostic confidence. 10,11 Yet, no simplified pipeline exists that enables a physician to efficiently leverage the complementary diagnostic video information provided by a multimodal bronchoscopic airway exam.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Synchronization and analysis of multimodal bronchoscopic airway exams for early lung cancer detection

Chang,

Daneshpajooh,

Byrnes

et al. 2024

Medical Imaging 2024: Image-Guided Procedures, Robotic Interventions, and Modeling

View full text Add to dashboard Cite

Because lung cancer is the leading cause of cancer-related deaths globally, early disease detection is vital. To help with this issue, advances in bronchoscopy have brought about three complementary noninvasive video modalities for imaging early-stage bronchial lesions along the airway walls: white-light bronchoscopy (WLB), autofluorescence bronchoscopy (AFB), and narrow-band imaging (NBI). Recent research indicates that performing a multimodal airway exam -i.e., using the three modalities together -potentially enables a more robust disease assessment than any single modality. Unfortunately, to perform a multimodal exam, the physician must manually examine each modality's video stream separately and then mentally correlate lesion observations. This process is not only extremely tedious and skill-dependent, but also poses the risk of missed lesions, thereby reducing diagnostic confidence. What is needed is a methodology and set of tools for easily leveraging the complementary information offered by these modalities. To address this need, we propose a framework for video synchronization and fusion tailored to multimodal bronchoscopic airway examination. Our framework, built into an interactive graphical system, entails a three-step process. First, for each of the three airway exams performed with a given bronchoscopic modality, several key airway video-frame landmarks are noted with respect to the patient's CT-based 3D airway tree model (CT = computed tomography), where the airway tree model serves as a reference space for the entire process. These landmarks create a set of connections between the videos and the airway tree to facilitate subsequent fine registration. Second, the landmark set, along with a set of additional video frames, which either contain detected lesions flagged by two deep-learning-based detection networks or lie between landmarks to help fill surface gaps, are finely registered to the airway tree, using a CT-video-based global registration method. Lastly, the registered frames are mapped and fused, via texture mapping, to the CT-based 3D airway tree's endoluminal surface. This enables sequential revising of synchronized multimodal surface structure and lesion locations through interactive graphical tools along a path navigating the airway tree. Results with patient multimodal bronchoscopic airway exams show the promise of our methods.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Frames Located Between the Landmark Set To Help Fill Endolum...mentioning

confidence: 99%

Section: Frames Located Between the Landmark Set To Help Fill Endolum...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Synchronization and analysis of multimodal bronchoscopic airway exams for early lung cancer detection

Chang,

Daneshpajooh,

Byrnes

et al. 2024

Medical Imaging 2024: Image-Guided Procedures, Robotic Interventions, and Modeling

View full text Add to dashboard Cite

show abstract

Self-supervised monocular depth and ego-motion estimation for CT-bronchoscopy fusion

Chang,

Higgins

2024

Medical Imaging 2024: Image-Guided Procedures, Robotic Interventions, and Modeling

View full text Add to dashboard Cite

The management of lung cancer necessitates robust diagnostic tools, with three-dimensional (3D) computed tomography (CT) imaging and bronchoscopy standing as pivotal complementary resources. Bronchoscopy captures live endobronchial video, providing striking detail of the airway tree's interior, while 3D CT scans contribute extensive anatomical knowledge. A significant gap persists, however, in linking these data-rich sources, such as in the fusion of video data from bronchoscopic airway exams and airway surface data from 3D CT scans. The main issue is the difficulty in simultaneously acquiring depth and camera pose information for bronchoscopic video frames. A solution to this problem can facilitate CT-video fusion/rendering, multimodal registration, and 3D cancer lesion localization. Deep-learning networks have been recently employed to estimate the depth and ego-motion information. Unfortunately, it is challenging to acquire the required training data, consisting of ground-truth pairs of bronchoscopic video frames and corresponding depth maps. Along this line, generative adversarial networks (GANs) have shown promise in domain transformation from CT-based endoluminal surface views into synthesized bronchoscopic frames. These synthesized views are consequently aligned with their CT-derived depth map, generating valuable training data. Nonetheless, such domain transformation techniques fail to utilize frame sequence knowledge and supply no information about the camera's ego-motion. Parallel studies in other domains, such as endoscopy, have emphasized the photometric consistency between adjacent frames to jointly offer depth and ego-motion estimation. Nevertheless, the texture-less and smooth endoluminal surface inside the airway restricts the generation of distinct depth maps with enhanced clarity and detail. To address this problem, we present a self-supervised training strategy that incorporates both domain transformation and photometric consistency for the Monodepth2 deep learning architecture, improving the depth and ego-motion prediction of bronchoscopic video frames. Results drawing on well-registered test data illustrate that the proposed strategy achieves clear and precise prediction. In addition, effective reference scaling factors are summarized from the test dataset, enabling real-world applications, such as 3D surface reconstruction, camera trajectory generation, and fusion between CT and bronchoscopic video.

show abstract

Bronchoscopic video synchronization for interactive multimodal inspection of bronchial lesions

Cited by 2 publications

References 0 publications

Synchronization and analysis of multimodal bronchoscopic airway exams for early lung cancer detection

Synchronization and analysis of multimodal bronchoscopic airway exams for early lung cancer detection

Self-supervised monocular depth and ego-motion estimation for CT-bronchoscopy fusion

Contact Info

Product

Resources

About