A deep learning framework for real‐time 3D model registration in robot‐assisted laparoscopic surgery

Abstract

Introduction The current study presents a deep learning framework to determine, in real‐time, position and rotation of a target organ from an endoscopic video. These inferred data are used to overlay the 3D model of patient's organ over its real counterpart. The resulting augmented video flow is streamed back to the surgeon as a support during laparoscopic robot‐assisted procedures. Methods This framework exploits semantic segmentation and, thereafter, two techniques, based on Convolutional Neural Networks and motion analysis, were used to infer the rotation. Results The segmentation shows optimal accuracies, with a mean IoU score greater than 80% in all tests. Different performance levels are obtained for rotation, depending on the surgical procedure. Discussion Even if the presented methodology has various degrees of precision depending on the testing scenario, this work sets the first step for the adoption of deep learning and augmented reality to generalise the automatic registration process.

A scoping review of deep learning in cancer nursing combined with augmented reality: The era of intelligent nursing is coming

“…In the preoperative evaluation, semi-automatic detection, recognition of specific patterns or segmentation of organs and lesions, is applied to preoperative imaging through typical DL tasks, and then the computer-generated 3D model is superimposed on the anatomical structure of real patients to solve the limited vision and depth information through AR so that DL ​+ ​AR can help doctors to eliminate dynamic interference and perform surgery in real-time and intuitively, thus improving the accuracy of surgery as well as reducing the incidence of surgery and complications. 43 , 44 Five studies 37 , 38 , 39 , 40 , 41 have proved that AI microscope can be used in efficient intelligent diagnosing, screening, and classifying a variety of cancer cells. It is confirmed that the ability of DL to present three-dimensional scenes with continuous depth sensation has a profound impact on AR, 45 so the innovative application of DL algorithm can also promote the better realization of AR.…”

Evolution of Surgical Robot Systems Enhanced by Artificial Intelligence: A Review

“…Numerous research teams have delved deeply into areas such as image segmentation, object recognition and tracking, pathological classification, complex signal processing, and multi-source information fusion using traditional ML and DL methods. [112,113] AI has significantly improved the accuracy of image segmentation, which is crucial for enhancing the objectivity and automation level of surgeries. Given the higher complexity of medical images compared to natural images, traditional segmentation methods, such as thresholding, region growing, and max-flow techniques, offer limited precision.…”

“…Numerous research teams have delved deeply into areas such as image segmentation, object recognition and tracking, pathological classification, complex signal processing, and multi‐source information fusion using traditional ML and DL methods. [ 112,113 ]…”

Deep Learning Based Image Processing for Robot Assisted Surgery: A Systematic Literature Survey

“…Eight out of ten studies used accuracy as a performance measuring metric along with others. The workflow recognition also overlaps with the future state prediction and phase recognition [222], [225]. The Table 11 contains the other necessary and relevant details about the section of the study.…”