Deep Sequential Mosaicking of Fetoscopic Videos

Abstract

Twin-to-twin transfusion syndrome treatment requires fetoscopic laser photocoagulation of placental vascular anastomoses to regulate blood flow to both fetuses. Limited field-of-view (FoV) and low visual quality during fetoscopy make it challenging to identify all vascular connections. Mosaicking can align multiple overlapping images to generate an image with increased FoV, however, existing techniques apply poorly to fetoscopy due to the low visual quality, texture paucity, and hence fail in longer sequences due to the drift accumulated over time. Deep learning techniques can facilitate in overcoming these challenges. Therefore, we present a new generalized Deep Sequential Mosaicking (DSM) framework for fetoscopic videos captured from different settings such as simulation, phantom, and real environments. DSM extends an existing deep image-based homography model to sequential data by proposing controlled data augmentation and outlier rejection methods. Unlike existing methods, DSM can handle visual variations due to specular highlights and reflection across adjacent frames, hence reducing the accumulated drift. We perform experimental validation and comparison using 5 diverse fetoscopic videos to demonstrate the robustness of our framework.

Inter-foetus Membrane Segmentation for TTTS Using Adversarial Networks

“…The proposed approach may also be integrated with recent work, which deals with vessel segmentation from placenta images 1,34 , stitching of fetoscopy images to build placental panoramic image 12,44 and classification of TTTS surgical phases 38 .…”

Deep learning-based fetoscopic mosaicking for field-of-view expansion

“…The recorded videos are converted to frames using the FFmpeg software. 2 To extract a square frame from the circular (Fig. 4), in order to use it as the input for the proposed network, we detect the circular mask of the scope through pixel-based image threshold and morphology.…”

“…Automatic selection of occlusion-free fetoscopic video segments [4] can help in identifying relevant segments for mosaicking. We showed in [2] that deep learning-based image alignment helps in reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution.…”

“…Mosaicking for the FoV expansion in fetoscopy has been explored using feature-based, intensity-based, and deep learning-based methods [2,3,10,21,22,26,27]. Reeff et al [22] and Daga et al [10] used the classical image featurebased matching method for creating mosaics from planar placenta images.…”

“…We perform comparison of the proposed fetoscopic video mosaicking (FVM) framework with existing methods using 5 diverse datasets. This paper is an extended version of the work presented at the MICCAI 2019 conference [2] and provides a broader insight into the fetoscopic mosaicking research, comprehensive analysis of both qualitative and quantitative results and comparison with the existing methods.…”