First‐in‐human real‐time AI‐assisted instrument deocclusion during augmented reality robotic surgery

Abstract

The integration of Augmented Reality (AR) into daily surgical practice is withheld by the correct registration of pre‐operative data. This includes intelligent 3D model superposition whilst simultaneously handling real and virtual occlusions caused by the AR overlay. Occlusions can negatively impact surgical safety and as such deteriorate rather than improve surgical care. Robotic surgery is particularly suited to tackle these integration challenges in a stepwise approach as the robotic console allows for different inputs to be displayed in parallel to the surgeon. Nevertheless, real‐time de‐occlusion requires extensive computational resources which further complicates clinical integration. This work tackles the problem of instrument occlusion and presents, to the authors’ best knowledge, the first‐in‐human on edge deployment of a real‐time binary segmentation pipeline during three robot‐assisted surgeries: partial nephrectomy, migrated endovascular stent removal, and liver metastasectomy. To this end, a state‐of‐the‐art real‐time segmentation and 3D model pipeline was implemented and presented to the surgeon during live surgery. The pipeline allows real‐time binary segmentation of 37 non‐organic surgical items, which are never occluded during AR. The application features real‐time manual 3D model manipulation for correct soft tissue alignment. The proposed pipeline can contribute towards surgical safety, ergonomics, and acceptance of AR in minimally invasive surgery.

A System for Mixed-Reality Holographic Overlays of Real-Time Rendered 3D-Reconstructed Imaging Using a Video Pass-through Head-Mounted Display—A Pathway to Future Navigation in Chest Wall Surgery

“…It is also conceivable that solutions for the automated overlay of imaging through previously trained AI-based registration models will also be introduced into the application in the near future. In robotic-assisted procedures, recently published AI models in the field of surgical computer vision may contribute to realizing navigational imaging overlays through real-time de-occlusion of surgical instruments [36].…”

Privacy-proof Live Surgery Streaming

“…Furthermore, this instrument occlusion sensitivity can be addressed by real-time deep-learning surgical instruments detection, which can be performed on the same computer. 21,22 The high-performance, high-cost computer used is too heavy for the application at hand. Smaller dedicated on-edge computational devices are now entering the market and are expected to decrease costs four-fold.…”

Evolution of Three-Dimensional Computed Tomography Imaging in Thoracic Surgery

“…Real-time surgical guidance is one of the goals for surgical navigation and has recently been tried in various fields of surgery. In 2023, first-in-human real-time AI-assisted instrument deocclusion during AR robotic surgery was reported [ 68 ]. Furthermore, the same group conducted first-in-human AI-assisted AR robotic lung surgery in two lower lobectomies with RATS [ 69 ].…”