Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance

Abstract

Purpose The current study aimed to propose a Deep Learning (DL) and Augmented Reality (AR) based solution for a in-vivo robot-assisted radical prostatectomy (RARP), to improve the precision of a published work from our group. We implemented a two-steps automatic system to align a 3D virtual ad-hoc model of a patient’s organ with its 2D endoscopic image, to assist surgeons during the procedure. Methods This approach was carried out using a Convolutional Neural Network (CNN) based structure for semantic segmentation and a subsequent elaboration of the obtained output, which produced the needed parameters for attaching the 3D model. We used a dataset obtained from 5 endoscopic videos (A, B, C, D, E), selected and tagged by our team’s specialists. We then evaluated the most performing couple of segmentation architecture and neural network and tested the overlay performances. Results U-Net stood out as the most effecting architectures for segmentation. ResNet and MobileNet obtained similar Intersection over Unit (IoU) results but MobileNet was able to elaborate almost twice operations per seconds. This segmentation technique outperformed the results from the former work, obtaining an average IoU for the catheter of 0.894 (σ = 0.076) compared to 0.339 (σ = 0.195). This modifications lead to an improvement also in the 3D overlay performances, in particular in the Euclidean Distance between the predicted and actual model’s anchor point, from 12.569 (σ= 4.456) to 4.160 (σ = 1.448) and in the Geodesic Distance between the predicted and actual model’s rotations, from 0.266 (σ = 0.131) to 0.169 (σ = 0.073). Conclusion This work is a further step through the adoption of DL and AR in the surgery domain. In future works, we will overcome the limits of this approach and finally improve every step of the surgical procedure.

A deep learning framework for real‐time 3D model registration in robot‐assisted laparoscopic surgery

“…Indeed, the rotation on the Z-axis is irrelevant because of the catheter's symmetry, while the rotation along the Y-axis can be successfully retrieved from the semantic map, 16 as proven by results in Table 3. Thus, only the rotation along the X-axis should be predicted.…”

“…The reason why the rotation can be more easily retrieved, is that, among the three axes, only one is effectively unknown. Indeed, the rotation on the Z ‐axis is irrelevant because of the catheter's symmetry, while the rotation along the Y ‐axis can be successfully retrieved from the semantic map, 16 as proven by results in Table 3. Thus, only the rotation along the X ‐axis should be predicted. When the organ is the only information to fall back on, different problems arise, such as organs' deformability, difficulty in distinguishing organ's texture from the surrounding tissues, and the fact that their rotation is bound to a fixed anatomical constraint.…”

“…Its coordinate is assigned to the centre of mass of the target's 3D model to perform the anchoring and the real‐time superimposition translation in the frame space. The same ellipse is used to infer the scale factor of the frame, meaning the distance from the camera, as its dimension is proportional to the one of the organ that is fitting 16 …”

“…Although this problem has been studied for many years, the advent of DL gave a renewed push to find a solution, as well as it fostered the research in other fields of application, such as the medical 14 , 15 , 16 or face recognition 17 , 18 domains. Old pose estimation methods were based on geometrical approaches, as for example, Feature‐based methods, 19 which tried to establish correspondences between 3D models and 2D images of objects by using manually annotated local features.…”

Vision Transformer for femur fracture classification

“…This idea was demonstrated in a previous work [6] from our research group, where a deep learning [7] based method was used to classify femur fractures and the performance of physicians with and without its help was compared. Deep learning is becoming more and more widely used, giving astonishing results in different fields of application, such as surgery [8,9,10] and face recognition [11]. In the vision domain, after the introduction of AlexNet [12] on the ImageNet competition in 2017, the applications of Convolutional Neural Networks (CNNs) have been increasing for their ability to capture the spatial dependencies in an image.…”