FetNet: a recurrent convolutional network for occlusion identification in fetoscopic videos

Abstract

Purpose Fetoscopic laser photocoagulation is a minimally invasive surgery for the treatment of twin-to-twin transfusion syndrome (TTTS). By using a lens/fibre-optic scope, inserted into the amniotic cavity, the abnormal placental vascular anastomoses are identified and ablated to regulate blood flow to both fetuses. Limited field-of-view, occlusions due to fetus presence and low visibility make it difficult to identify all vascular anastomoses. Automatic computer-assisted techniques may provide better understanding of the anatomical structure during surgery for risk-free laser photocoagulation and may facilitate in improving mosaics from fetoscopic videos. Methods We propose FetNet, a combined convolutional neural network (CNN) and long short-term memory (LSTM) recurrent neural network architecture for the spatio-temporal identification of fetoscopic events. We adapt an existing CNN architecture for spatial feature extraction and integrated it with the LSTM network for end-to-end spatio-temporal inference. We introduce differential learning rates during the model training to effectively utilising the pre-trained CNN weights. This may support computer-assisted interventions (CAI) during fetoscopic laser photocoagulation. Results We perform quantitative evaluation of our method using 7 in vivo fetoscopic videos captured from different human TTTS cases. The total duration of these videos was 5551 s (138,780 frames). To test the robustness of the proposed approach, we perform 7-fold cross-validation where each video is treated as a hold-out or test set and training is performed using the remaining videos. Conclusion FetNet achieved superior performance compared to the existing CNN-based methods and provided improved inference because of the spatio-temporal information modelling. Online testing of FetNet, using a Tesla V100-DGXS-32GB GPU, achieved a frame rate of 114 fps. These results show that our method could potentially provide a real-time solution for CAI and automating occlusion and photocoagulation identification during fetoscopic procedures.

Deep learning-based fetoscopic mosaicking for field-of-view expansion

“…Mosaicking from fetoscopic videos particularly remains challenging due to fetoscopic device limitations (monocular low-resolution fetoscopic camera with FoV limitation), occlusion by the fetus, ablation tool presence and occasional bleeding, non-planar views, turbid amniotic fluid, specular highlights and reflection due to variation in the light source, distortion due to light refraction [ 9 ], and texture paucity. Automatic selection of occlusion-free fetoscopic video segments [ 4 ] can help in identifying relevant segments for mosaicking. We showed in [ 2 ] that deep learning-based image alignment helps in reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution.…”

A shape-constraint adversarial framework with instance-normalized spatio-temporal features for inter-fetal membrane segmentation

“…However, for 29 frames (15.62%), the colour and the texture of the background were similar to those of the membrane, yielding to the false-positive segmentation. This limitation could be solved by a preliminary frame-selection step (Bano et al, 2020), where only frames in which the membrane is visible are further processed by the proposed framework. A possible limitation of the proposed framework can be seen in segmentation inaccuracies in clips with rapid and wide movements of the fetoscope, where the membrane changes in appearance very quickly.…”

“…Another possible limitation may be seen when processing frames with membrane occlusions due to fetal movements, umbilical cord and glare from the photocoagulation laser. While we did not address this specific aspect in this paper, a possible solution to tackle it would be to rely on the automatic selection of occlusion-free frames (Bano et al, 2020).…”

“…Work relevant to TTTS video analysis focuses on surgical planning, surgicalphase detection, intrauterine cavity segmentation, placental vessel segmentation and mosaicking reconstruction. Examples of surgical-phase detection in TTTS include the work of Vasconcelos et al (2018), where a ResNet encoder is used to detect the ablation phase, and Bano et al (2020), which extends Vasconcelos et al (2018) by adding an LSTM layer to integrate temporal information and detect different surgical phases. In Torrents- Barrena et al (2020), a reinforcement learning approach that relies on capsule networks has been proposed to perform automatic intrauterine cavity segmentation from multi-planar placenta magnetic-resonance imaging recordings, for surgical planning purposes.…”

FetReg: Placental Vessel Segmentation and Registration in Fetoscopy Challenge Dataset

“…This is because of the presence of fetuses, laser ablation fibre and working channel port which can occlude the field-of-view of the fetoscope. Frame registration and mosaicking is only required in occlusion-free video segments that capture the surface of the placenta [28] as these are the segments in which the surgeon is exploring the intraoperative environment to identify abnormal vascular connections. Expanding the field-of-view through mosaicking in these video segments can facilitate the procedure by providing better visualization of the environment.…”