Study on augmented reality for robotic surgery bedside assistants

Abstract

Robotic surgery bedside assistants play an important role in robotic procedures by performing intra-corporeal tasks while accommodating the physical presence of the robot. We hypothesized that an augmented reality headset enabling 3D intracorporeal vision while facing the surgical field could decrease time and improve accuracy of robotic bedside tasks. Bedside assistants (one physician assistant, one medical student, three surgical trainees, and two attending surgeons) performed validated tasks within a mock abdominal cavity with a surgical robot docked. Tasks were performed with a bedside monitor providing 2D or 3D vision, or an optical see-through head-mounted augmented reality device with 2D or 3D vision. The effect of augmented reality device resolution on performance was also evaluated. For the simplest task of touching a straw, performance was generally high, regardless of mode of visualization. With more complex tasks, including stapling and pulling a ring along a path, 3D augmented reality decreased time and number of errors per task. 3D augmented reality allowed the physician assistant to perform at the level of an attending surgeon using 3D augmented reality (p = 0.08). All participants had improved times for the ring path task with better resolution (lower resolution 23 ± 11 s vs higher resolution 14 ± 4 s, p = 0.002). 3D augmented reality vision with high resolution decreased time and improved accuracy of more complex tasks, enabling a less experienced robotic surgical bedside assistant to function similar to attending surgeons. These data warrant further study with additional complex tasks and bedside assistants at various levels of training.

Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery

“…General applications ( ), general surgery ( ), neurosurgery ( ), robot-assisted surgery ( ), and vascular surgery ( ) were the next most prevalent. Robot-assisted surgery applications included orthopedic applications [ 55 ], general bedside tasks [ 56 ], and endoluminal interventions [ 57 ]. Papers focused on general surgery and associated applications included telementoring [ 58 ], teleproctoring [ 59 ], surgical navigation [ 60 ], and liver resection [ 61 ].…”

“…Telestrations, such as virtual arrows and other virtual annotations were common in telementoring and teleproctoring [ 58 , 59 , 60 ]. Intraoperative endoscope video is the primary real-time data source for laparoscopic and endoscopic procedures and robot-assisted surgery [ 56 , 82 ]. Cone-beam CT employs a similar imaging strategy as CT and improves on some limitations of X-ray fluoroscopy, mainly the 2D nature of the data, by providing the capacity for a 3D snapshot of the intraoperative patient anatomy [ 84 , 85 ].…”

“…Other 3D information that was included alongside the surface rendered models included relevant preoperative planning information, such as locations of pedicle screws [ 47 ], target anatomical contours in maxillofacial surgery [ 54 , 90 ], and tissue deformation models in liver resection surgery [ 61 ]. Several papers mentioned the streaming of raw 3D data directly to the OST-HMD for visualization, including intraoperative stereo endoscope video for robotic bedside task support [ 56 ], flexible endoscope steering [ 91 ], and endoluminal interventions [ 57 ]. Unlike 3D surface models, volume-rendered 3D models do not require manual segmentation and preprocessing to create and instead rely on discrete sampling of a 3D dataset.…”

“…The remaining articles ( ) involving adjacent overlay of virtual content centered on tasks which would typically rely on a computer monitor for data display and aimed to improve access to the relevant data. Use-cases involving adjacent overlay included the display of stereo endoscope video for robotic bedside assistants [ 56 ] and fluoroscopic imaging data [ 81 ] in the field of view of the user but adjacent to the patient.…”

“…With OST-HMD use, it is possible to position virtually augmented content in a comfortable location based on user preference. Despite the additional weight and bulk of wearing an HMD throughout a procedure, ergonomic benefits were reported in OST-HMD use for data display during robotic surgery bedside assistant tasks [ 56 ] and in live fluoroscopic imaging display during orthopedic interventions [ 81 ]. However, the contribution of OST-HMD use to ergonomics was not all positive; challenges with optical display quality and latency can contribute to nausea and eye strain [ 110 ].…”