Head-Mounted Display-Based Augmented Reality for Image-Guided Media Delivery to the Heart: A Preliminary Investigation of Perceptual Accuracy

Abstract

By aligning virtual augmentations with real objects, optical see-through head-mounted display (OST-HMD)-based augmented reality (AR) can enhance user-task performance. Our goal was to compare the perceptual accuracy of several visualization paradigms involving an adjacent monitor, or the Microsoft HoloLens 2 OST-HMD, in a targeted task, as well as to assess the feasibility of displaying imaging-derived virtual models aligned with the injured porcine heart. With 10 participants, we performed a user study to quantify and compare the accuracy, speed, and subjective workload of each paradigm in the completion of a point-and-trace task that simulated surgical targeting. To demonstrate the clinical potential of our system, we assessed its use for the visualization of magnetic resonance imaging (MRI)-based anatomical models, aligned with the surgically exposed heart in a motion-arrested open-chest porcine model. Using the HoloLens 2 with alignment of the ground truth target and our display calibration method, users were able to achieve submillimeter accuracy (0.98 mm) and required 1.42 min for calibration in the point-and-trace task. In the porcine study, we observed good spatial agreement between the MRI-models and target surgical site. The use of an OST-HMD led to improved perceptual accuracy and task-completion times in a simulated targeting task.

Augmenting Performance: A Systematic Review of Optical See-Through Head-Mounted Displays in Surgery

“…Laparoscopic surgery applications included cholecystectomy [ 72 ] and general laparoscopic surgery [ 73 ]. Heart surgery related applications included guiding the targeted delivery of media to the surface of the infarcted heart in regenerative medicine [ 33 ]. Otolaryngology applications included median neck and brachial cyst excision [ 74 ] and neck cancer [ 75 ].…”

“…There were several papers which reported the use of custom markers for surface-based tracking, though the performance did not significantly differ from ArUco or Vuforia tracking solutions [ 82 , 97 ]. In the surface-based tracking strategy, the square marker fiducials served to localize the patient and any tracked tools relative to the reference frame of the OST-HMD, allowing for the overlay of virtual content [ 33 , 46 , 88 , 91 , 92 ]. Other groups reported the use of retroreflective spheres and the infrared camera capabilities on the HoloLens 1, HoloLens 2, or custom devices for improved tracking accuracy and performance over monocular RGB led tracking [ 60 , 75 , 90 , 98 ].…”

“…The most common strategy for OST-HMD system evaluation included the use of an anthropomorphic phantom ( )—a reasonable approach, given the early stage investigational nature of many OST-HMD-based navigation platforms. Phantom models were used for assessment of error in orthopedic screw insertion [ 46 , 76 , 93 ], K-Wire placement for arthroplasty [ 50 , 51 ], maxillofacial drilling [ 54 , 69 ], head and neck cancer navigation [ 75 ], and as a means of simulating targeted media delivery in cardiac regenerative medicine [ 33 ]. Phantom models are relatively inexpensive and provide an excellent and safe platform for the initial quantitative assessment of a guidance system.…”

“…Animal models improve on the anatomical relevance of cadaver models due to the incorporation of realistic tissue deformation and real-time respiratory and cardiac motion. Animal models were investigated for OST-HMD-led percutaneous tumor puncture [ 68 ], placement of radioactive seeds during brachytherapy [ 98 ], and media delivery in targeted cardiac therapies [ 33 ].…”

“…Simultaneous localization and mapping allows for HMD localization within the world; however, it does not provide the same accuracy as marker-based localization and is subject to drift caused by error propagation over time [ 32 ]. Though the accuracy and stability of SLAM is not enough to rely on for precise tasks, it can be invaluable as an alternative method to maintain alignment of virtual entities to real objects during instances where marker-based localization is unstable or lost [ 33 ].…”