Using spatial-temporal ensembles of convolutional neural networks for lumen segmentation in ureteroscopy

Abstract

Purpose Ureteroscopy is an efficient endoscopic minimally invasive technique for the diagnosis and treatment of upper tract urothelial carcinoma. During ureteroscopy, the automatic segmentation of the hollow lumen is of primary importance, since it indicates the path that the endoscope should follow. In order to obtain an accurate segmentation of the hollow lumen, this paper presents an automatic method based on convolutional neural networks (CNNs). Methods The proposed method is based on an ensemble of 4 parallel CNNs to simultaneously process single and multi-frame information. Of these, two architectures are taken as core-models, namely U-Net based in residual blocks ($$m_1$$ m 1 ) and Mask-RCNN ($$m_2$$ m 2 ), which are fed with single still-frames I(t). The other two models ($$M_1$$ M 1 , $$M_2$$ M 2 ) are modifications of the former ones consisting on the addition of a stage which makes use of 3D convolutions to process temporal information. $$M_1$$ M 1 , $$M_2$$ M 2 are fed with triplets of frames ($$I(t-1)$$ I ( t - 1 ) , I(t), $$I(t+1)$$ I ( t + 1 ) ) to produce the segmentation for I(t). Results The proposed method was evaluated using a custom dataset of 11 videos (2673 frames) which were collected and manually annotated from 6 patients. We obtain a Dice similarity coefficient of 0.80, outperforming previous state-of-the-art methods. Conclusion The obtained results show that spatial-temporal information can be effectively exploited by the ensemble model to improve hollow lumen segmentation in ureteroscopic images. The method is effective also in the presence of poor visibility, occasional bleeding, or specular reflections.

A Novel Preoperative Prediction Model Based on Deep Learning to Predict Neoplasm T Staging and Grading in Patients with Upper Tract Urothelial Carcinoma

“…However, deep learning applications in UTUC are rare at the present stage. Lazo et al proposed a spatial-temporal ensemble of convolutional neural networks for lumen segmentation to identify UTUC during ureteroscopy [ 41 ]. As an exploration of ureteroscopic image recognition, this technology is still far from clinical application.…”

Autonomous Intraluminal Navigation of a Soft Robot using Deep-Learning-based Visual Servoing

“…The lumen center detection module consists of two steps, a lumen segmentation stage and a center detection algorithm. The first part, the lumen segmentation step, consists of an ensemble of CNNs based on [19] and depicted in Fig. 3.…”

“…This might be related to the fact that the dataset in which it was tested does not contain challenging cases. The ensemble model, which obtained the best performances and in previous work has shown to be the more robust against conditions variability and artifacts [19], was chosen to be implemented in the visual servoing module.…”

“…Convolutional Neural Networks (CNNs) on the other hand tend to generalize better when they are trained in a large enough amount of data [3]. In this regard, we propose a CNN based on a model previously validated on patient image data [19], adapting it to a lighter version to be implemented in a robotic device.…”

Automatic hands-free visualization of a six degrees of freedom agent within a complex anatomical space

“…The Visual-Servoing Module aims at enabling autonomous guidance having the tip of the catheter always pointing towards the centre of the ureter. A deep learning based visual servoing high-level controller is used to autonomously segment the lumen from the camera images, as presented in [188], and compute the centre of the ureter. The information from the detected centre is used to calculate the error and bring the tip of the endoscope towards the detected centre point.…”