FoldIt: Haustral Folds Detection and Segmentation in Colonoscopy Videos

CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy

“…To test the effectiveness of RT-GAN in fold segmentation context (indicative of the total missed surface during colonoscopy), we added RT-GAN on top of FoldIt haustral fold frame-based model [16]. In Figure 2, we compare RT-GAN, FoldIt, TempCycleGAN, and RecycleGAN results on public video sequences from Ma et al [14].…”

“…In contrast, CycleGAN [36] presented a method for unsupervised/unpaired domain translation using a cycle consistency loss for forward and backward translation between two given domains. A number of follow-up works have adopted this cycle consistency loss or variations of it for task-specific frame-based image-toimage domain translation models [11], [16], [34]. The taskspecific components in these frame-based models are critical for obtaining best results but these need to be redesigned or dropped altogether when transitioning to a completely new video-to-video domain translation model.…”

“…In total, training dataset is composed of 1500 frame triplets, while the validation and testing datasets were composed of 900 and 600 frame triplets respectively. Haustral fold segmentation data is generated in a similar manner to Mathew et al [16].…”

“…In contrast, RT-GAN builds on the representations learned by any unsupervised image-to-image domain translation model and adds temporal consistency to these without needing to redesign task-specific components. We demonstrate the effectiveness of RT-GAN in adding lightweight temporal consistency to two frame-based models, FoldIt [16] haustral fold segmentation model with some inherent "quasi-consistency" across frames and CLTS-GAN color-lighting-texture-specular reflection augmentation model with no consistency at all across frames. The contributions of this work are as follows: 1) A method to add temporal consistency to established frame-based models 2) An efficient and less resource-hungry method for video domain translation 3) A tunable way to control the trade off between temporal smoothness and faithfulness to the frame-based model 4) Demonstrate the effectiveness of RT-GAN in two challenging use cases: haustral fold segmentation (indicative of missed surface) and real colonoscopy video simulation.…”

Development of endoscopic SLAM and the challenges

“…For instance, in the context of colonoscopy, the haustral folds found in the intestine possess consistent edges, rendering them a suitable substitute for features. Recent research has examined the feasibility of utilizing haustral folds as landmarks for Endoscopic SLAM, where semantic segmentation using GAN has been employed for this purpose [29].…”