Autonomous Endoscope Robot Positioning Using Instrument Segmentation With Virtual Reality Visualization

Abstract

This paper presents a method for endoscope's autonomous positioning by a robotic endoscope holder for minimally invasive surgery. The method improves human-robot cooperation in robot-assisted surgery by allowing the endoscope holder to acknowledge the surgeon's view projection and navigate the camera without manual control. The real-time prediction of next desired camera location is estimated using segmented instrument's tip locations from endoscope video and surgeon's attention focus given by tracked virtual reality headset. To tackle the issue of real-time surgical instrument segmentation for more precise instrument tip localization, we propose the YOLOv3 and ResNet Combined Neural Network. The method showed an 86.6% IoU across MICCAI'17 Endovis datasets with 30 frames per second processing speed. The proposed pipeline was implemented in ROS on Ubuntu with visualization running under Windows operating system in Unity3D. The simulation demonstrates the robotic arm, endoscope, and surgical environment visualized in 3D in the virtual reality headset to provide a stable view of the endoscope and improve the surgeon's perception of the operating environment.INDEX TERMS autonomous robot control, artificial neural network, minimally invasive surgery, object segmentation, human-robot cooperation.

Robotic Endoscope Control Via Autonomous Instrument Tracking

“…A starting point for an autonomous REC strategy is to reposition the endoscope so as to keep the surgical instrument tip centered in the view. Such an approach could work already when only the 2D position of the instrument in the endoscopic image is available ( Uecker et al, 1995 ; Osa et al, 2010 ; Agustinos et al, 2014 ; Zinchenko and Song, 2021 ). In this case, the endoscope zoom level (depth) is left uncontrolled.…”

“…Some of these methods also require a 3D geometrical instrument model ( Agustinos et al, 2014 ), limiting the flexibility of the system. Approaches such as proposed by Zinchenko and Song (2021) have also suggested to replace the endoscope screen with a head-mounted virtual reality device that facilitates the estimation of the surgeon’s attention focus from the headset’s gyroscope. In this scenario, the autonomous REC strategy aims to reposition the endoscope with the aim of maintaining the weighted center of mass between the instruments’ contour centers and the point of focus in the center of the view.…”

Endoscope Automation Framework with Hierarchical Control and Interactive Perception for Multi-Tool Tracking in Minimally Invasive Surgery

“…Hence, the development and automation of robot-assisted endoscope control systems are considered crucial steps for improving both the efficacy and safety of MIS [ 4 ]. Recent works in RAMIS worked on replacing the assistant by designing interfaces the robotic systems that allow to control the endoscope direct hand-operation [ 5 ], foot pedal [ 6 ], eye-tracking [ 7 ] or head-mount displays [ 8 , 9 ], etc. Nevertheless, these control schemes require specialized training and might result in higher mental load on the surgeons [ 3 ].…”

“…As an alternative, YOLO-based detectors are increasingly employed [ 16 ]. Some research has advocated for specialized detectors that focus on generating image masks or precise tool poses [ 8 , 17 , 18 , 19 ]. Despite promising results, these methods are contingent on data availability and require frequent re-training for new surgical tools.…”

“…Most methods focus on single-tool tracking or employ data-driven methodologies [ 15 , 24 ]. Some even integrate surgeon preferences explicitly or via additional interfaces [ 8 , 25 , 26 ]. However, the extension of these methods for multi-tool handling is still an open research area.…”