Architecture of a Hybrid Video/Optical See-through Head-Mounted Display-Based Augmented Reality Surgical Navigation Platform

Abstract

In the context of image-guided surgery, augmented reality (AR) represents a ground-breaking enticing improvement, mostly when paired with wearability in the case of open surgery. Commercially available AR head-mounted displays (HMDs), designed for general purposes, are increasingly used outside their indications to develop surgical guidance applications with the ambition to demonstrate the potential of AR in surgery. The applications proposed in the literature underline the hunger for AR-guidance in the surgical room together with the limitations that hinder commercial HMDs from being the answer to such a need. The medical domain demands specifically developed devices that address, together with ergonomics, the achievement of surgical accuracy objectives and compliance with medical device regulations. In the framework of an EU Horizon2020 project, a hybrid video and optical see-through augmented reality headset paired with a software architecture, both specifically designed to be seamlessly integrated into the surgical workflow, has been developed. In this paper, the overall architecture of the system is described. The developed AR HMD surgical navigation platform was positively tested on seven patients to aid the surgeon while performing Le Fort 1 osteotomy in cranio-maxillofacial surgery, demonstrating the value of the hybrid approach and the safety and usability of the navigation platform.

Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery

“…Other 3D information that was included alongside the surface rendered models included relevant preoperative planning information, such as locations of pedicle screws [ 47 ], target anatomical contours in maxillofacial surgery [ 54 , 90 ], and tissue deformation models in liver resection surgery [ 61 ]. Several papers mentioned the streaming of raw 3D data directly to the OST-HMD for visualization, including intraoperative stereo endoscope video for robotic bedside task support [ 56 ], flexible endoscope steering [ 91 ], and endoluminal interventions [ 57 ].…”

“…In the surface-based tracking strategy, the square marker fiducials served to localize the patient and any tracked tools relative to the reference frame of the OST-HMD, allowing for the overlay of virtual content [ 33 , 46 , 88 , 91 , 92 ]. Other groups reported the use of retroreflective spheres and the infrared camera capabilities on the HoloLens 1, HoloLens 2, or custom devices for improved tracking accuracy and performance over monocular RGB led tracking [ 60 , 75 , 90 , 98 ]. Moving away from the reliance on constructed marker configurations, several papers reported the use of point-based marker-less tracking for virtual content overlay [ 48 , 50 , 76 ].…”

“…Evaluation in patients primarily involved early feasibility assessments which included surgeon feedback and usability considerations, and typically the virtual information was not used to inform treatment decisions [ 46 , 62 , 63 , 72 , 96 ]. Several papers investigated the quantitative accuracy of OST-HMD led navigation in preliminary patient studies, including assessment of orthopedic pedicle screw navigation [ 47 ], neuronavigation [ 65 ], maxillofacial osteotomy tracing [ 90 ], and maxillofacial tumor resection [ 52 ].…”

Preclinical Application of Augmented Reality in Pediatric Craniofacial Surgery: An Accuracy Study

“…These included, but were not limited to, registration errors and the small augmentable field of view. Therefore, whereas there is a more stringent requirement for accuracy (i.e., sub-millimetric precision), other “surgery-specific” devices should be addressed, such as the ones previously described in the literature [ 17 , 38 , 39 , 40 , 41 ].…”

Bioengineering, augmented reality, and robotic surgery in vascular surgery: A literature review

“…Eight papers out of 11, report the use of HMDs or Smart Glasses (11)(12)(13)(14)(15)(16)(17)(18). These displays provide the user with an egocentric viewpoint and allow operators to work hands-free, and according to recent literature (22,23), they have been deemed the most ergonomic solution for applications including manual tasks performed by the user under direct vision, like what happens in open surgery. However, as is evident in this study, there is growing interest in the literature on the use of this type of display even for minimally invasive procedures, such as endovascular procedures.…”