Retrieval and registration of long-range overlapping frames for scalable mosaicking of in vivo fetoscopy

Abstract

PurposeThe standard clinical treatment of Twin-to-Twin transfusion syndrome consists in the photo-coagulation of undesired anastomoses located on the placenta which are responsible to a blood transfer between the two twins. While being the standard of care procedure, fetoscopy suffers from a limited field-of-view of the placenta resulting in missed anastomoses. To facilitate the task of the clinician, building a global map of the placenta providing a larger overview of the vascular network is highly desired.MethodsTo overcome the challenging visual conditions inherent to in vivo sequences (low contrast, obstructions or presence of artifacts, among others), we propose the following contributions: (1) robust pairwise registration is achieved by aligning the orientation of the image gradients, and (2) difficulties regarding long-range consistency (e.g. due to the presence of outliers) is tackled via a bag-of-word strategy, which identifies overlapping frames of the sequence to be registered regardless of their respective location in time.ResultsIn addition to visual difficulties, in vivo sequences are characterised by the intrinsic absence of gold standard. We present mosaics motivating qualitatively our methodological choices and demonstrating their promising aspect. We also demonstrate semi-quantitatively, via visual inspection of registration results, the efficacy of our registration approach in comparison with two standard baselines.ConclusionThis paper proposes the first approach for the construction of mosaics of placenta in in vivo fetoscopy sequences. Robustness to visual challenges during registration and long-range temporal consistency are proposed, offering first positive results on in vivo data for which standard mosaicking techniques are not applicable.Electronic supplementary materialThe online version of this article (10.1007/s11548-018-1728-4) contains supplementary material, which is available to authorized users.

Deep learning-based fetoscopic mosaicking for field-of-view expansion

“…Mosaicking for the FoV expansion in fetoscopy has been explored using feature-based, intensity-based, and deep learning-based methods [2,3,10,21,22,26,27]. Reeff et al [22] and Daga et al [10] used the classical image featurebased matching method for creating mosaics from planar placenta images.…”

“…However, current clinical regulations and limited form factor of the fetoscope hinder the use of such a tracker in intraoperative settings. Peter et al [21] proposed a direct pixel-wise alignment of gradient orientations for creating a mosaic from a single in vivo fetoscopic video. Gaisser et al [15] detected stable regions on veins of the placenta using a region-based convolutional neural network and then used these detected regions as features for placental image registration in an underwater phantom setting [15].…”

“…A mosaic is generated by aligning the relative transformations between the pair of consecutive fetoscopic images with respect to a reference frame. A small error in the relative transformations can introduce large drift in the mosaic, where global consistency alignment techniques and use of electromagnetic tracker can help to minimize the drifting error [ 21 , 26 , 27 ]. Tella-Amo et al [ 26 , 27 ] assumed placenta to be planar and static and integrated the electromagnetic tracker with the fetoscopic in a synthetic and ex vivo setup to propose a mosaicking approach capable of handling the drifting error.…”

Pruning strategies for efficient online globally consistent mosaicking in fetoscopy

“…• Despite the image is obtained using a fetoscope, we use a pinhole camera model. 13,15 • The EMT field is not perfect. Yet, we assume we are working on the center of the field, and that therefore, that the distortions due to inhomogeneities in the electromagnetic field are neglibible.…”

“…12 consists of matching exclusively the images that are highly likely to share visual corresponding points, avoiding incoherent attempts. A proposed technique for overlap detection in monocular systems is to use a bag of words 13 (BoW), which creates a dictionary of visual words and assigns a signature per image. This signature summarizes the number of occurrences of the visual words in a histogram.…”